batch.spec

python = <<EOF

from batchutil import bazzinga

def foo():
    return f"foo: I am {self.name}"

def bar():
    return f"bar: I am {self.name}"
    
def baz():
    return bazzinga(self)
    
EOF

<batch>
  name p&l_calulator
  exec_str echo %foo()%
</batch>

batchutil.py

def bazzinga(batch):
    return f"baz: I am {batch.name}"

test.py

from apacheconfig import *
  
class BatchJobSpec:
    
    def __init__(self, pythoncode , batch):
        self.pythoncode = pythoncode
        self.name = batch.get("name")
        self._exec_str = batch.get("exec_str")
        self.exec()
    
    @property
    def exec_str(self):
        _exec = self._exec_str.split("%")
        
        for i in range(len(_exec)):
            if _exec[i].endswith('()'):
                f = self.__getattribute__(_exec[i][:-2])
                _exec[i] = f()
                return ''.join(_exec)
            
        
    def exec(self):
        locals_ = {'self': self}
        exec(self.pythoncode, locals_)
        
        _exec = []
        for attrib in locals_.keys():
            if not attrib in ['__builtins__', 'self']:
                print('__setattr__({})'.format(attrib))
                self.__setattr__(attrib, locals_[attrib])

with make_loader() as loader:
    config = loader.load('batch.spec')

batchjobspec = BatchJobSpec(config.get("python"), config.get("batch"))
print(batchjobspec.pythoncode)
print(batchjobspec.name)
print(batchjobspec.exec_str)
print(batchjobspec.baz())

Questions:

1. I want "batchjobspec.exec_str" to return "echo foo: I am p&l_calulator"
2. Note that I have also created batchutil.py, this will have shared functions that can be used across batchspec files. Can I load functions inside batchutil.py in the local context of batchjobspec too? This is needed because functions inside util may want to refer some of batchjobspec specific attributes. For example: function "bazzinga()" inside batchutil is referring to self.name.
3. Is self.name a right way to access batchjobspec attributes inside functions of batchutil? Is there some other better way to do this?

@agrawalo

First change def exec(self) to:

    def exec(self):
        locals_ = {'self': self}
        exec(self.pythoncode, locals_)
        
        _exec = []
        for attrib in locals_.keys():
            if not attrib in ['__builtins__', 'self']:
                print('__setattr__({})'.format(attrib))
                self.__setattr__(attrib, locals_[attrib])

Note: You have to verify in your environment if ['__builtins__', 'self'] are the only stopwords to use.
The result of loacal_.keys() ist Python Version dependant.

Line 9: This can be deleted, as no longer needed.

@agrawalo

Note that batchutil.py will have shared functions that can be used in many batchspec files.

This will not work, as you can't reach self in batchutil.py.

Can I load batchutil.py in the local context too? Note that method "bazzinga()" inside batchutil referring to self.name

Yes, you have to add self:

def bazzinga(self):
    return f"baz: I am {self.name}"

batchjobspec.bazzinga = <imported from batchutil.py>.bazzinga

@agrawalo

I want the print statement below to return me "echo foo: I am p&l_calulator"

print(batchjobspec.exec_str)

Add to __init__:

    def __init__(self, pythoncode , batch):
        self._batch = batch

Line 8: self.exec_str ... could be delete as we handel the whole batch

Add class @property def batch(self):

    @property
    def batch(self):

1. Split exec string by '%'

        _exec = self._batch['exec'].split('%')
   

2. Find function to replace

      for i in range(len(_exec)):
          if _exec[i].endswith('()'):
   

3. Get the reference of the function namend _exec[i][:-2]

                f = self.__getattribute__(_exec[i][:-2])
   

4. Call the function and replace with the returned str

                _exec[i] = f()
   

5. Return the whole _exec as one str

           return ''.join(_exec)
   

Usage

print(batchjobspec.batch)

Updated incorporating comments from above.

Now batchjobspec.bar() throws following exception

---

NameError Traceback (most recent call last)
in
32 print(batchjobspec.name)
33 print(batchjobspec.exec_str)
---> 34 print(batchjobspec.baz())

in baz()

~/apacheconfig/batchutil.py in bazzinga()
1 def bazzinga(self):
----> 2 return f"baz: I am {self.name}"

NameError: name 'self' is not defined

Questions:
2. Note that I have also created batchutil.py, this will have shared functions that can be used across batchspec files. Can I load functions inside batchutil.py in the local context of batchjobspec too? This is needed because functions inside util may want to refer some of batchjobspec specific attributes. For example: function "bazzinga()" inside batchutil is referring to self.name.

I feel that from batchutil import bazzinga inside batch.spec will not work.
Change return bazzinga() to return bazzinga(self.name) to make it independent from self.

3. Is self.name a right way to access batchjobspec attributes inside functions of batchutil?
   Is there some other better way to do this?

Using self.name is the natural pythonic way and there are no side effects like using global ....

In batch.spec

line

"def foo():
return f"foo: I am {self.name}""

still looks weird to me because visually self is sort of not available in this block. To make it bit more intuitive I guess changing this to something like below will look good and intuitive.

"def foo(batch):
return f"foo: I am {batch.name}""

this will result in changing following line

"exec_str echo %foo()%"

to

"exec_str echo %foo(self)%"

is this possible?

@agrawalo

In batch.spec
still looks weird to me because self is sort of not available in this block. To make it bit more intuitive I guess changing this to something like below will look good and intuitive.

"def foo(batch):
return f"foo: I am {batch.name}""

I recommend the following, as it becomes a indirect member of the class and writing def function_name(self) are common.

def foo(self):
    return f"foo: I am {self.name}""

This needs to change the following line in def exec(self):

                _exec[i] = f(self)

The drawback are, you can't call batchjobspec.baz() any more you have to use batchjobspec.baz(batchjobspec).
You have to decide, more logic in batch.spec or in Python code itself.
And second, you have to use this with all def ... in batch.spec even if not needed.

this will result in changing following line
"exec_str echo %foo(self)%"

Yes, but this complicate finding function parameter in exec string

I actually made it a staticmethod. Please review the code https://gist.github.com/agrawalo/407caf4dcd44687fae6989f46b54975a