eisenmp_examples package

Subpackages

Submodules

eisenmp_examples.cmd module

commandline menu for eisenmp_examples

ToDo manifest del __pycache__ flask collision with existing version in eisenradio rename http server to url.py start browser auto

write docu and user menu pypi in color:

python -m eisenmp_examples.cmd  commandline menu
python -m eisenmp_examples.url  ajax http server
eisenmp_examples.cmd.main()

Call menu_main to start the module from command line.

eisenmp_examples.cmd.menu_main()

Main menu to choose from.

eisenmp_examples.eisenmp_exa_bruteforce module

Generator module, Worker in /worker folder

Mandatory: - import eisenmp - worker must have the ‘toolbox’ (kwargs) as arg in entry function (arg count 1) - path to worker module and the entry function reference, all are str

class eisenmp_examples.eisenmp_exa_bruteforce.ModuleConfiguration

Bases: object

Loading the worker module has nothing to do with name resolution in ‘this’ module.
We can load from network share.
We have full access to all queues and methods. eisenmp = eisenmp.Mp()
We can transfer as much variable values to the worker as we want. Smaller is better. Spawn start copies values.
bruteforce_module = {'worker_path': '/home/docs/checkouts/readthedocs.org/user_builds/eisenmp-examples/checkouts/latest/eisenmp_examples/worker/eisenmp_exa_wrk_bf_bruteforce.py', 'worker_ref': 'worker_entrance'}
lang_dicts_paths = ['.\\lang_dictionaries\\ger\\german.dic', '.\\lang_dictionaries\\eng\\words.txt']
lang_dicts_url_zip_dct = {'https://github.com/44xtc44/eisenmp_examples/raw/dev/eisenmp_examples/lang_dictionaries/eng/SCOWL-wl.zip': 'words.txt', 'https://github.com/44xtc44/eisenmp_examples/raw/dev/eisenmp_examples/lang_dictionaries/ger/german.zip': 'german.dic'}
list_reducer_module = {'worker_path': '/home/docs/checkouts/readthedocs.org/user_builds/eisenmp-examples/checkouts/latest/eisenmp_examples/worker/eisenmp_exa_wrk_bf_reduce.py', 'worker_ref': 'worker_entrance'}
eisenmp_examples.eisenmp_exa_bruteforce.extract_zip_wordlists()

File system unzip.

eisenmp_examples.eisenmp_exa_bruteforce.load_lang_word_dict()

Load dictionaries from disk or gitHub

eisenmp_examples.eisenmp_exa_bruteforce.main()
eisenmp_examples.eisenmp_exa_bruteforce.mp_brute_force(search_instance)

Generator Part

  • Worker 1 - only one worker is active at any time

Produce str permutations for len(str), if len(str) = 3 we have 3! = 6 permutations Have a string and two tools. (A) wordlist as a dict and (B) itertools permutation generator. Searched string is hiding in the permutations. Check if the permutation makes sense and compare it with the wordlist.

eisenmp_examples.eisenmp_exa_bruteforce.mp_reduce(search_instance)

  • Worker 2

Took all language word list and put them in a dict. Deleted all words with more or less characters.

Worker will now reduce the list generated from dict. Remove all words not matching characters and character count in search str from list. Worker puts result in output q.

eisenmp_examples.eisenmp_exa_bruteforce.rt_loader()

rainbow table test if dict is only hosted in parent proc, if spawn

eisenmp_examples.eisenmp_exa_bruteforce.start_raid()

  • Manager -

Decide raid tactics. Brute force on smaller strings. List reduce attempt for larger strings.

Grab config values and push it to the two possible functions.

    1. len(String) <= 10, combined brute force dictionary attack.

    1. len(String) >= 11, Reduce a dictionary len(str) condensed list and count characters.

See mp_brute_force and mp_reduce for more info.

eisenmp_examples.eisenmp_exa_bruteforce.start_raid_sequential()

Blocks creation of a new instance, else next instance starts while previous instances runs

eisenmp_examples.eisenmp_exa_bruteforce.word_dicts_file_system_load()

  • Entry ZIP FS

eisenmp_examples.eisenmp_exa_bruteforce.word_dicts_git_hub_load()

  • Entry ZIP load URL

eisenmp_examples.eisenmp_exa_bruteforce.wordlist_download(downloader)

Store response object in downloader instance. We could show dots during download, or open in binary, or csv, or ….

eisenmp_examples.eisenmp_exa_bruteforce.wordlists_in_memory(downloader)

Open zip archive and load one file ‘zipped_filename’ Return as text.

eisenmp_examples.eisenmp_exa_bruteforce.worker_module_set(function_ref)

which worker module and function to load

eisenmp_examples.eisenmp_exa_double_q module

Template for Manager

  • Watchdog Thread included, shows pid of its proc

  • double feeder

  • custom queues and queues in category

  • fake production of audio and video coding, is loop through to result dict

  • two processors work on different streams, batches, (src is same here, so what)

class eisenmp_examples.eisenmp_exa_double_q.ModuleConfiguration

Bases: object

More advanced template. Multiprocess ‘spawn’ in ‘ProcEnv’ to work with all OS. - toolbox.kwargs shows all avail. vars and dead references of dicts, lists, instances, read only

template_module = {'WORKER_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/eisenmp-examples/checkouts/latest/eisenmp_examples/worker/eisenmp_exa_wrk_double.py', 'WORKER_REF': 'worker_entrance'}
watchdog_module = {'WORKER_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/eisenmp-examples/checkouts/latest/worker/eisenmp_exa_wrk_watchdog.py', 'WORKER_REF': 'mp_start_show_threads'}
eisenmp_examples.eisenmp_exa_double_q.audio_generator_batch_1()
eisenmp_examples.eisenmp_exa_double_q.audio_generator_batch_7()
eisenmp_examples.eisenmp_exa_double_q.main()
eisenmp_examples.eisenmp_exa_double_q.manager_entry()

  • Generator - One time execution.

Divide workload between processes / CPU -

eisenmp_examples.eisenmp_exa_double_q.video_generator_batch_1()
eisenmp_examples.eisenmp_exa_double_q.video_generator_batch_7()

eisenmp_examples.eisenmp_exa_each_flask_orm_srv_one_cpu module

See some examples, it’s always the same process, you extend or switch off defaults

class eisenmp_examples.eisenmp_exa_each_flask_orm_srv_one_cpu.ModuleConfiguration

Bases: object

You can use the class to have your variables available in the module.

dir_name = '/home/docs/checkouts/readthedocs.org/user_builds/eisenmp-examples/checkouts/latest/eisenmp_examples'
first_module = {'WORKER_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/eisenmp-examples/checkouts/latest/eisenmp_examples/worker/eisenmp_exa_wrk_flask_orm_srv_one_cpu.py', 'WORKER_REF': 'worker'}
foo = {'WORKER_PATH': 'bar', 'WORKER_REF': 'baz'}
eisenmp_examples.eisenmp_exa_each_flask_orm_srv_one_cpu.main()
eisenmp_examples.eisenmp_exa_each_flask_orm_srv_one_cpu.manager()

  • Manager -

!!! Database must be [created with one process only], then many procs can read, write !!!

eisenmp_examples.eisenmp_exa_ghettorecorder module

GhettoRecorder command line app instances are distributed over two processes. See the documentation images.

The parent process is controlling the instances via a child process thread manager.
Three queues are assigned to each child process. In, out, audio Queue.

A Watchdog Thread is included, it shows the pid of its proc.

class eisenmp_examples.eisenmp_exa_ghettorecorder.ModuleConfiguration(proc_max=3, app_max=3, proc_balance=True)

Bases: object

More advanced template. Multiprocess ‘spawn’ in ‘ProcEnv’ to work with all OS. - toolbox.kwargs shows all avail. vars and dead references of dicts, lists, instances, read only

watchdog_module = {'WORKER_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/eisenmp-examples/checkouts/latest/worker/eisenmp_exa_wrk_watchdog.py', 'WORKER_REF': 'mp_start_show_threads'}
worker_module = {'WORKER_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/eisenmp-examples/checkouts/latest/eisenmp_examples/worker/eisenmp_exa_wrk_ghettorecorder.py', 'WORKER_REF': 'worker_entrance'}
eisenmp_examples.eisenmp_exa_ghettorecorder.com_out_collector_parent(emp)

Analyse tuples and lists. (A) Tuple of three is command output strings (radio, command, result).

(B) Lists have information tuples in it. Tuple of two is info dict (radio, dict). Each radio collects header info and status.

eisenmp_examples.eisenmp_exa_ghettorecorder.create_parent_qs(emp)

Create multiple input qs, one output q and one audio output q.

Store qs in eisenmp instance to inform worker procs.
Store qs in modConf instance user dict to attach user frontend (FE). FE connect qs and pulls info from dict.
eisenmp_examples.eisenmp_exa_ghettorecorder.create_threads(emp)

Put commands from frontend in input q. Collect output info of all app instances.

eisenmp_examples.eisenmp_exa_ghettorecorder.frontend_entry()

Return dict for frontend. Connect qs and read content-type.

eisenmp_examples.eisenmp_exa_ghettorecorder.input_collector_parent(emp)

Frontend sends commands to parent process input q. Input q forks to com_in process queues.

Function analyses command tuples.
Each app instance is stored with its process number.
eisenmp_examples.eisenmp_exa_ghettorecorder.input_create_recorder(emp, tup, q_dict_key, a_dct)

Request instance creation of recorder.

Params:

emp: eisenmp instance, controls the process creation and provide default qs

Params:

tup: tuple with arguments for recorder instance creation

Params:

q_dict_key: modConf.q_dict_key[q_dict_key] dictionary, key name to access dict with qs references

Params:

a_dct: custom dict with eisenmp and modConf.user_dict key values, prevent mass variable usage

eisenmp_examples.eisenmp_exa_ghettorecorder.input_put_command(emp, tup, q_dict_key)

Distribute commands:

('nachtflug','exec', 'setattr(self,"runs_listen",True) to process, app.
Params:

emp: eisenmp instance, controls the process creation and provide default qs

Params:

tup: tuple with arguments for recorder instance creation

Params:

q_dict_key: modConf.q_dict_key[q_dict_key] dictionary, key name to access dict with qs references

eisenmp_examples.eisenmp_exa_ghettorecorder.input_put_function(emp, tup, q_dict_key, a_dct)

Each app instance is stored with its process number. User can send commands to correct process, app.

Balanced means putting next app instance into next process number. Else stack until upper limit hits.

Params:

emp: eisenmp instance, controls the process creation and provide default qs

Params:

tup: tuple with arguments for recorder instance creation

Params:

q_dict_key: modConf.q_dict_key[q_dict_key] dictionary, key name to access dict with qs references

Params:

a_dct: custom dict with eisenmp and modConf.user_dict key values, prevent mass variable usage

eisenmp_examples.eisenmp_exa_ghettorecorder.input_shutdown_procs(emp, tup, q_dict_key, a_dct)

Shutdown all or requested processes.

Params:

emp: eisenmp instance, controls the process creation and provide default qs

Params:

tup: tuple with arguments for recorder instance creation

Params:

q_dict_key: modConf.q_dict_key[q_dict_key] dictionary, key name to access dict with qs references

Params:

a_dct: custom dict with eisenmp and modConf.user_dict key values, prevent mass variable usage

eisenmp_examples.eisenmp_exa_ghettorecorder.main()
eisenmp_examples.eisenmp_exa_ghettorecorder.manager_entry()

Run this module. | Must take category for qs. queue_cust_dict_category_create. | Access in worker toolbox.ghetto_qs[‘com_in_1’] | Note: Can not access in worker toolbox[‘com_in_1’], only toolbox.com_in_1 | Run Test on Manager: | create_radio = (‘create’, ‘nachtflug’, ‘http://85.195.88.149:11810’, ‘home/osboxes/abracadabra’) | emp.queue_cust_dict_cat[‘child_qs’][‘com_in_1’].put(create_radio)

eisenmp_examples.eisenmp_exa_ghettorecorder.run_ghetto_example()

eisenmp_examples.eisenmp_exa_ghettorecorder_cnct module

class eisenmp_examples.eisenmp_exa_ghettorecorder_cnct.Helper

Bases: object

cmd_lst = [('nachtflug', 'exec', 'setattr(self,"runs_listen", True)')]
create_lst = [('create', 'nachtflug', 'http://85.195.88.149:11810', 'recorder', True), ('create', 'hirschmilch', 'https://hirschmilch.de:7001/prog-house.mp3', 'recorder'), ('create', 'zenstyle', 'https://radio4.cdm-radio.com:18004/stream-mp3-Zen', 'recorder'), ('create', 'classic_ro', 'http://37.251.146.169:8000/streamHD', 'recorder', True), ('create', 'yeahmon', 'http://c3.radioboss.fm:8095/autodj', 'recorder', True), ('create', 'aacchill', 'http://radio4.vip-radios.fm:8020/stream128k-AAC-Chill_autodj', 'recorder'), ('create', 'playurban', 'http://live.playradio.org:9090/UrbanHD', 'recorder')]
eisenmp_examples.eisenmp_exa_ghettorecorder_cnct.kill_all_processes()

shutdown_process

eisenmp_examples.eisenmp_exa_ghettorecorder_cnct.main()

Frontend sends commands as tuple to input q. Input q distributes to processes worker, worker distributes to radio instance. Frontend connects to audio q.

eisenmp_examples.eisenmp_exa_ghettorecorder_cnct.start_recorder()
eisenmp_examples.eisenmp_exa_ghettorecorder_cnct.stop_recorder()

eisenmp_examples.eisenmp_exa_http module

class eisenmp_examples.eisenmp_exa_http.ModuleConfiguration

Bases: object

You can use the class to have your variables available in the module.

‘worker_http()’ is executed by all processes on a CPU somewhere. Each ‘worker_http()’ gets (one by one) a list chunk from eisenmp.mp_input_q.

dir_name = '/home/docs/checkouts/readthedocs.org/user_builds/eisenmp-examples/checkouts/latest/eisenmp_examples'
first_module = {'WORKER_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/eisenmp-examples/checkouts/latest/eisenmp_examples/eisenmp_exa_http.py', 'WORKER_REF': 'worker_http'}
foo = {'WORKER_PATH': 'bar', 'WORKER_REF': 'baz'}
watchdog_module = {'WORKER_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/eisenmp-examples/checkouts/latest/eisenmp_examples/worker/eisenmp_watchdog.py', 'WORKER_REF': 'mp_start_show_threads'}
class eisenmp_examples.eisenmp_exa_http.MyServer(request, client_address, server)

Bases: BaseHTTPRequestHandler

do_GET()
toolbox = None
eisenmp_examples.eisenmp_exa_http.main()
eisenmp_examples.eisenmp_exa_http.manager_http_srv()

  • Manager -

eisenmp_examples.eisenmp_exa_http.worker_http(toolbox)

  • Worker -

Blocked, no loop here toolbox is the all-in-one box for vars and queues. incl. ModuleConfiguration We have PROCS_MAX, we have ports to add, we need worker in groups to serve ports

eisenmp_examples.eisenmp_exa_multi_srv_each_cpu module

See some examples, it’s always the same process, you extend or switch off defaults

Multiple server on each CPU core, if server has that many cores. needs Flask_SQLAlchemy_Project_Template >=1.3

All worker are thread started -> and can not send ‘return False’ (run_forever()), no stop confirmation of module_loader mod of eisenmp avail

class eisenmp_examples.eisenmp_exa_multi_srv_each_cpu.ModuleConfiguration

Bases: object

You can use the class to have your variables available in the module.

dir_name = '/home/docs/checkouts/readthedocs.org/user_builds/eisenmp-examples/checkouts/latest/eisenmp_examples'
first_module = {'WORKER_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/eisenmp-examples/checkouts/latest/eisenmp_examples/worker/eisenmp_exa_wrk_multi_srv_each_cpu.py', 'WORKER_REF': 'worker'}
foo = {'WORKER_PATH': 'bar', 'WORKER_REF': 'baz'}
eisenmp_examples.eisenmp_exa_multi_srv_each_cpu.main()
eisenmp_examples.eisenmp_exa_multi_srv_each_cpu.manager()

  • Manager -

!!! Database must be [created with one process only], then many procs can read, write !!!

ORM https://en.wikipedia.org/wiki/Object%E2%80%93relational_mapping

eisenmp_examples.eisenmp_exa_prime module

Multiprocessor Prime number finder (worker) and number Generator.

Example for a two in one module. Manager and worker.

This is NOT ADVISABLE and only to show the mechanics. Use a separate worker module to not import your ‘Generator’ imports twice. That is what the loader is for.

class eisenmp_examples.eisenmp_exa_prime.ModuleConfiguration

Bases: object

You can use the class to have your variables available in the module.

‘worker_prime()’ is executed by all processes on a CPU somewhere. Each ‘worker_prime()’ gets (one by one) a list chunk from eisenmp.mp_input_q. You have full access to all queues and methods. mp = eisenmp.Mp()

first_module = {'WORKER_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/eisenmp-examples/checkouts/latest/eisenmp_examples/eisenmp_exa_prime.py', 'WORKER_REF': 'worker_prime'}
foo = {'WORKER_PATH': 'bar', 'WORKER_REF': 'baz'}
eisenmp_examples.eisenmp_exa_prime.calc_prime(toolbox)

Calc prime num.

The stop message is detected in ‘workload_get’, in this example.

eisenmp_examples.eisenmp_exa_prime.generator_prime()

Manager - One time execution. Exits if generator is empty.

eisenmp_examples.eisenmp_exa_prime.is_prime(n: int) → bool

https://en.wikipedia.org/wiki/Primality_test

mod by 44xtc44, limit = int(math.sqrt(n)) , isqrt(n) is Python 3.8, package shall run 3.7

eisenmp_examples.eisenmp_exa_prime.main()
eisenmp_examples.eisenmp_exa_prime.number_generator()

Generates numbers from start count. Has an end value, range.

eisenmp_examples.eisenmp_exa_prime.remove_header(toolbox)

Transport ticket with consecutive number. Remove if no recreation of order is necessary.

eisenmp_examples.eisenmp_exa_prime.send_eta_data(toolbox)

list of [PERF_HEADER_ETA, PERF_CURRENT_ETA] to ProcInfo, to calc arrival time ETA

eisenmp_examples.eisenmp_exa_prime.send_output(toolbox, prime_lst)

Put your findings in the output list. Find results in the ‘eisenmp_utils.Result.result_dict’

Params:

toolbox: -

Params:

prime_lst: findings of prime num calc

eisenmp_examples.eisenmp_exa_prime.worker_prime(toolbox)

  • WORKER - Called with a single arg (name it xyz) in a loop until returns False.

Start, Entry, Exit of this ‘single’ process worker. We return True to get next list chunk, whatever object is in the rows. Fed from mp_input_q to our toolbox. toolbox is your work instance with queues, messages, list chunk, and work tools like language dictionary or hash list.

toolbox.foo, gives also access to all attributes and values of the ‘modConf.foo’ instance, you have created

eisenmp_examples.eisenmp_exa_prime.workload_get(toolbox)

eisenmp_examples.eisenmp_exa_web_csv module

Green CSV generator / Worker in folder /worker

Download a NZ government report and calculate the average of split lists. See URL. Try your own URL. They have a lot.

class eisenmp_examples.eisenmp_exa_web_csv.ModuleConfiguration

Bases: object

Fill the existing attributes. You can use the class to have your variables available in the module. You have full access to all queues and methods. mp = eisenmp.Mp()

dl_url = 'https://www.stats.govt.nz/assets/Uploads/International-trade/International-trade-December-2022-quarter/Download-data/international-trade-december-2022-quarter.zip'
first_module = {'worker_path': '/home/docs/checkouts/readthedocs.org/user_builds/eisenmp-examples/checkouts/latest/eisenmp_examples/worker/eisenmp_exa_wrk_csv.py', 'worker_ref': 'worker_entrance'}
watchdog_module = {'WORKER_PATH': '/home/docs/checkouts/readthedocs.org/user_builds/eisenmp-examples/checkouts/latest/eisenmp_examples/worker/eisenmp_exa_wrk_watchdog.py', 'WORKER_REF': 'mp_start_show_threads'}
eisenmp_examples.eisenmp_exa_web_csv.download_report_zip_archive(emp, report_file)
eisenmp_examples.eisenmp_exa_web_csv.g_in_mem_csv(downloader)
eisenmp_examples.eisenmp_exa_web_csv.g_use_fs_csv(report_file)
eisenmp_examples.eisenmp_exa_web_csv.generator_calc_csv()

  • Manager -

Generator part of calc CSV. Worker module in /worker folder.

Params:

result_lbl: find result in a list; sum findings, mandatory

Params:

use_file_system: True ‘use_file_system’; False: in mem unzip

Params:

url: dl_url, if not ‘use_file_system’, URL of csv file

Params:

zipped_filename: ‘revised.csv’, name of the uncompressed file in zip archive

Params:

csv_col_name: ‘value’, # table column header name, mandatory

Params:

‘ROWS_MAX’: num list rows to calc in one loop for each process

Params:

‘num_proc’: procs to start, can be more or less than system CPU core count

eisenmp_examples.eisenmp_exa_web_csv.main()
eisenmp_examples.eisenmp_exa_web_csv.origin_msg_create()
eisenmp_examples.eisenmp_exa_web_csv.unzip_on_file_system(downloader, report_f)

eisenmp_examples.url module

class eisenmp_examples.url.Menu

Bases: object

exa_tpl_lst = [(0, 'Multiple server in each process/CPU core - share a port range - share a DB. <br>Worker Green, Yellow and Blue', <function main>, 'True'), (1, 'Prime Number calculation', <function main>, 'False'), (2, 'Every Flask server in a different process - share a port range - share a DB. <br>Worker Yellow and Blue', <function main>, 'True'), (3, 'Web CSV large list. Average for each chunk of a large list column to simply calc the results later.', <function main>, 'False'), (4, 'One simple http server presents a radio on every process. "Unemployed" worker exit.', <function main>, 'True'), (5, 'Each process has two Queues feed audio and video. Merge in a fake production.', <function main>, 'False'), (6, 'Brute force attack with dictionary and itertools generator', <function main>, 'False'), (7, 'Distribute recorder app instances over different processes.', <function main>, 'False'), (8, 'start radio recorder', <function start_recorder>, 'False'), (9, 'stop radio recorder', <function stop_recorder>, 'False'), (10, 'kill all processes', <function kill_all_processes>, 'False')]
example_menu = ['Multiple server in each process/CPU core - share a port range - share a DB. <br>Worker Green, Yellow and Blue', 'Prime Number calculation', 'Every Flask server in a different process - share a port range - share a DB. <br>Worker Yellow and Blue', 'Web CSV large list. Average for each chunk of a large list column to simply calc the results later.', 'One simple http server presents a radio on every process. "Unemployed" worker exit.', 'Each process has two Queues feed audio and video. Merge in a fake production.', 'Brute force attack with dictionary and itertools generator', 'Distribute recorder app instances over different processes.', 'start radio recorder', 'stop radio recorder', 'kill all processes']
class eisenmp_examples.url.MyServer(request, client_address, server)

Bases: BaseHTTPRequestHandler

do_GET()
do_POST()
eisenmp_examples.url.main()
eisenmp_examples.url.run_http(com_queue=None)

Blocked, no loop here

Params:

com_queue: test setup, know when server is up

eisenmp_examples.z module

Module contents