IPython Parallel API

From IPRE Wiki
Revision as of 18:25, 22 March 2014 by Doug Blank (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Anatomy of a conversation between Hub, Engines, and Notebook.


Starting the hub:

$ ipcontroller
2014-03-22 07:59:06.383 [IPControllerApp] Using existing profile dir: u'/home/dblank/.ipython/profile_default'
2014-03-22 07:59:06.392 [IPControllerApp] Hub listening on tcp://127.0.0.1:38580 for registration.
2014-03-22 07:59:06.394 [IPControllerApp] Hub using DB backend: 'NoDB'
2014-03-22 07:59:06.647 [IPControllerApp] hub::created hub
2014-03-22 07:59:06.647 [IPControllerApp] writing connection info to /home/dblank/.ipython/profile_default/security/ipcontroller-client.json
2014-03-22 07:59:06.648 [IPControllerApp] writing connection info to /home/dblank/.ipython/profile_default/security/ipcontroller-engine.json
2014-03-22 07:59:06.649 [IPControllerApp] task::using Python leastload Task scheduler
2014-03-22 07:59:06.649 [IPControllerApp] Heartmonitor started
2014-03-22 07:59:06.660 [IPControllerApp] Creating pid file: /home/dblank/.ipython/profile_default/pid/ipcontroller.pid
send: ['<IDS|MSG>', 'ead801e1ac21830cd3fce6c9ca58e233740b00fc0de958f3f50704d15af67734', 
       '{"date":"2014-03-22T07:59:06.669259","username":"dblank","session":"f832cf19-0b90-4db4-a27d-7fafe243c96c","msg_id":"e230c872-2b5e-46cb-a038-37cb76eff5bd","msg_type":"connection_request"}', 
       '{}', 
       '{}', 
       '{}']
2014-03-22 07:59:06.669 [scheduler] Scheduler started [leastload]
2014-03-22 07:59:06.673 [IPControllerApp] client::client '\x00 \xaaF\x92' requested 'connection_request'
2014-03-22 07:59:06.674 [IPControllerApp] client::client ['\x00 \xaaF\x92'] connected
send: ['\x00 \xaaF\x92', '<IDS|MSG>', 'e962d83b0ca2762b621078b622045fb8a379235d5d8460327f68ef355708c62b', 
       '{"date":"2014-03-22T07:59:06.674472","username":"dblank","session":"8987776d-2efd-41f5-b142-71f8c19d29da","msg_id":"154e9aca-7ba9-4631-9c6e-1583d7af5a85","msg_type":"connection_reply"}', 
       '{"date":"2014-03-22T07:59:06.669259","username":"dblank","session":"f832cf19-0b90-4db4-a27d-7fafe243c96c","msg_id":"e230c872-2b5e-46cb-a038-37cb76eff5bd","msg_type":"connection_request"}', 
       '{}', 
       '{"status":"ok","engines":{}}']


This creates /home/dblank/.ipython/profile_default/security/ipcontroller-client.json:

{
  "control": 53346, 
  "task": 38578, 
  "notification": 44032, 
  "task_scheme": "leastload", 
  "mux": 36063, 
  "iopub": 58633, 
  "ssh": "", 
  "key": "81d89a45-f014-43bb-b13d-a15e38289b1f", 
  "registration": 38580, 
  "interface": "tcp://127.0.0.1", 
  "signature_scheme": "hmac-sha256", 
  "pack": "json", 
  "unpack": "json", 
  "location": "192.168.1.100"
}

This creates /home/dblank/.ipython/profile_default/security/ipcontroller-engine.json:

{
  "control": 34438, 
  "task": 58161, 
  "hb_ping": 40588, 
  "mux": 39251, 
  "pack": "json", 
  "hb_pong": 55458, 
  "ssh": "", 
  "key": "81d89a45-f014-43bb-b13d-a15e38289b1f", 
  "registration": 38580, 
  "interface": "tcp://127.0.0.1", 
  "iopub": 36488, 
  "signature_scheme": "hmac-sha256", 
  "unpack": "json", 
  "location": "192.168.1.100"
}

Starting an engine:

$ ipengine
2014-03-22 08:02:22.395 [IPEngineApp] Using existing profile dir: u'/home/dblank/.ipython/profile_default'
2014-03-22 08:02:22.396 [IPEngineApp] Loading url_file u'/home/dblank/.ipython/profile_default/security/ipcontroller-engine.json'
2014-03-22 08:02:22.403 [IPEngineApp] Registering with controller at tcp://127.0.0.1:38580
send: ['<IDS|MSG>', '119282d3ae19a304544d866a3559e062f51f38c782c2667a18d2233aad243150', 
       '{"date":"2014-03-22T08:02:22.405708","username":"dblank","session":"d035f477-971f-4a40-88ef-3386676900ab","msg_id":"23bc4db8-985a-4ead-b953-656ab7b906bd","msg_type":"registration_request"}', 
       '{}', 
       '{}', 
       '{"uuid":"d035f477-971f-4a40-88ef-3386676900ab"}']
2014-03-22 08:02:22.455 [IPEngineApp] Starting to monitor the heartbeat signal from the hub every 3010 ms.
2014-03-22 08:02:22.457 [IPEngineApp] Using existing profile dir: u'/home/dblank/.ipython/profile_default'
send: ['engine.0.status', '<IDS|MSG>', 'cbf29c12d95a7d781250500c5eee665f1055ac3f9fbc4ef200916cd610669d29', 
       '{"date":"2014-03-22T08:02:22.458492","username":"dblank","session":"d035f477-971f-4a40-88ef-3386676900ab","msg_id":"19d182d7-4c8d-4e1d-a688-8eed1298e0fe","msg_type":"status"}', 
       '{}', 
       '{}', 
       '{"execution_state":"starting"}']
2014-03-22 08:02:22.458 [IPEngineApp] Completed registration with id 0

Hub responds:

2014-03-22 08:02:22.407 [IPControllerApp] client::client 'd035f477-971f-4a40-88ef-3386676900ab' requested 'registration_request'
send: ['d035f477-971f-4a40-88ef-3386676900ab', '<IDS|MSG>', '70528c9e562a256b6d648d84244cf25de729d25ca55f2830448d7c2312002340', 
       '{"date":"2014-03-22T08:02:22.407932","username":"dblank","session":"8987776d-2efd-41f5-b142-71f8c19d29da","msg_id":"dcd814b1-da77-466e-baa7-6d8e1fe387ad","msg_type":"registration_reply"}', 
       '{}', 
       '{}', 
       '{"status":"ok","id":0,"hb_period":3000}']
2014-03-22 08:02:22.459 [IPControllerApp] WARNING | iopub::IOPub message lacks parent: {'parent_header': {}, 'msg_type': 'status', 'msg_id': '19d182d7-4c8d-4e1d-a688-8eed1298e0fe', 'content': {'execution_state': 'starting'}, 'header': {'date': datetime.datetime(2014, 3, 22, 8, 2, 22, 458492), 'username': 'dblank', 'session': 'd035f477-971f-4a40-88ef-3386676900ab', 'msg_id': '19d182d7-4c8d-4e1d-a688-8eed1298e0fe', 'msg_type': 'status'}, 'buffers': [], 'metadata': {}}
2014-03-22 08:02:27.652 [IPControllerApp] registration::finished registering engine 0:d035f477-971f-4a40-88ef-3386676900ab
send: ['<IDS|MSG>', '3e63f58b69d8826a46fe67c25eb4d0a1f94d789daaff519b3763111b3b2047b4', 
       '{"date":"2014-03-22T08:02:27.653050","username":"dblank","session":"8987776d-2efd-41f5-b142-71f8c19d29da","msg_id":"ed6abe6b-f2ee-4442-9a55-6e5d23fed420","msg_type":"registration_notification"}', 
       '{}', 
       '{}', 
       '{"uuid":"d035f477-971f-4a40-88ef-3386676900ab","id":0}']
2014-03-22 08:02:27.653 [IPControllerApp] engine::Engine Connected: 0

Starting a second engine:

$ ipengine
2014-03-22 08:04:58.921 [IPEngineApp] Using existing profile dir: u'/home/dblank/.ipython/profile_default'
2014-03-22 08:04:58.922 [IPEngineApp] Loading url_file u'/home/dblank/.ipython/profile_default/security/ipcontroller-engine.json'
2014-03-22 08:04:58.930 [IPEngineApp] Registering with controller at tcp://127.0.0.1:38580
send: ['<IDS|MSG>', 'dc27622c6c441bd50e0d25afd31b4d431b48fe62d7d15bf80470746ecd37877d', 
       '{"date":"2014-03-22T08:04:58.931313","username":"dblank","session":"0f017363-c43c-48bd-80f2-8c59790949c8","msg_id":"8e22c60a-b572-4408-bd19-e9fe1249c97d","msg_type":"registration_request"}', 
       '{}', 
       '{}', 
       '{"uuid":"0f017363-c43c-48bd-80f2-8c59790949c8"}']
2014-03-22 08:04:58.981 [IPEngineApp] Starting to monitor the heartbeat signal from the hub every 3010 ms.
2014-03-22 08:04:58.983 [IPEngineApp] Using existing profile dir: u'/home/dblank/.ipython/profile_default'
send: ['engine.1.status', '<IDS|MSG>', '56d5f199f5ba5434f4d4090767fa6553afeb14377285decd901f5c6bcb4ae2b4', 
       '{"date":"2014-03-22T08:04:58.984228","username":"dblank","session":"0f017363-c43c-48bd-80f2-8c59790949c8","msg_id":"10e95708-e80e-480c-ad03-2e89b5942200","msg_type":"status"}', 
       '{}', 
       '{}', 
       '{"execution_state":"starting"}']
2014-03-22 08:04:58.984 [IPEngineApp] Completed registration with id 1

Hub responds:

2014-03-22 08:04:58.931 [IPControllerApp] client::client '0f017363-c43c-48bd-80f2-8c59790949c8' requested 'registration_request'
send: ['0f017363-c43c-48bd-80f2-8c59790949c8', '<IDS|MSG>', 'a328d2cc6dc8eb31dbfb00d73f1f54934a4250d5ae238c7706df685368238909', 
       '{"date":"2014-03-22T08:04:58.932115","username":"dblank","session":"8987776d-2efd-41f5-b142-71f8c19d29da","msg_id":"301c816e-af65-4072-a668-3f4cf849f76e","msg_type":"registration_reply"}', 
       '{}', 
       '{}', 
       '{"status":"ok","id":1,"hb_period":3000}']
2014-03-22 08:04:58.985 [IPControllerApp] WARNING | iopub::IOPub message lacks parent: {'parent_header': {}, 'msg_type': 'status', 'msg_id': '10e95708-e80e-480c-ad03-2e89b5942200', 'content': {'execution_state': 'starting'}, 'header': {'date': datetime.datetime(2014, 3, 22, 8, 4, 58, 984228), 'username': 'dblank', 'session': '0f017363-c43c-48bd-80f2-8c59790949c8', 'msg_id': '10e95708-e80e-480c-ad03-2e89b5942200', 'msg_type': 'status'}, 'buffers': [], 'metadata': {}}
2014-03-22 08:05:03.652 [IPControllerApp] registration::finished registering engine 1:0f017363-c43c-48bd-80f2-8c59790949c8
send: ['<IDS|MSG>', '87a6468ad97d821d17097c412fb46127d6cca9a3f62d1c85ede4ffca67ef7e95', 
       '{"date":"2014-03-22T08:05:03.652914","username":"dblank","session":"8987776d-2efd-41f5-b142-71f8c19d29da","msg_id":"ea5a0790-9ea0-437a-b758-2bff658359f3","msg_type":"registration_notification"}', 
       '{}', 
       '{}', 
       '{"uuid":"0f017363-c43c-48bd-80f2-8c59790949c8","id":1}']
2014-03-22 08:05:03.653 [IPControllerApp] engine::Engine Connected: 1


Notebook Client():

send: ['<IDS|MSG>', '78898f44474e1066e1c1c522e818f9dcbe75bb50b55ce47285b909fc447100ae', 
       '{"date":"2014-03-22T08:35:14.929199","username":"dblank","session":"10e5b992-2cff-4e08-af3a-b28d55848856","msg_id":"7a94b006-4bc2-4de3-a3a4-bb0c32e0e912","msg_type":"connection_request"}', 
       '{}', 
       '{}', 
       '{}']
recv: ['7b83108eaa5e86d1ccfdee31e34bad847ea725868bb46679e0988102a04c02f7', 
       '{"date":"2014-03-22T08:35:14.930409","username":"dblank","session":"8987776d-2efd-41f5-b142-71f8c19d29da","msg_id":"6b075f75-57a6-4b20-b2f6-7c5ff731d163","msg_type":"connection_reply"}', 
       '{"date":"2014-03-22T08:35:14.929199","username":"dblank","session":"10e5b992-2cff-4e08-af3a-b28d55848856","msg_id":"7a94b006-4bc2-4de3-a3a4-bb0c32e0e912","msg_type":"connection_request"}', 
       '{}', 
       '{"status":"ok","engines":{"1":"0f017363-c43c-48bd-80f2-8c59790949c8","0":"d035f477-971f-4a40-88ef-3386676900ab"}}']


Notebook requests Client()[:].get("random.random()") to Hub: (use Client()[:].execute("random.random()") to send plain text to eval)

send: ['kernel.12afdd67-e27f-4778-802d-bfd2312924ae.pyin', '<IDS|MSG>', '40a43a0c523dd3d7779f1c487ab19e7783a9542945ea70ac9cf289c6b7b8980f', 
       '{"date":"2014-03-22T08:58:27.741018","username":"kernel","session":"b13bfb83-70d6-4bbe-bcfe-f51da8dbfe58","msg_id":"32122a72-665c-49d9-be0c-3e2721e15106","msg_type":"pyin"}',  
       '{"username":"username","msg_id":"7331A22A3B5C4ED69D4F529A4AED457F","msg_type":"execute_request","session":"709DCA0E6DE44B51B98C39D57CBB646C"}', 
       '{}', 
       '{"execution_count":17,"code":"view.get(\\"random.random()\\")"}']

Hub sends request to engines:

send: ['d035f477-971f-4a40-88ef-3386676900ab', '<IDS|MSG>', 'e0c3a882683c1966251a9b0c5e1203b811f5eb650b3ab03733a3be8291ee8841', 
       '{"date":"2014-03-22T08:58:27.744095","username":"dblank","session":"10e5b992-2cff-4e08-af3a-b28d55848856","msg_id":"1e2fe257-63d6-40a7-8a5e-9b3b28b4ddb8","msg_type":"apply_request"}', 
       '{}', 
       '{}', 
       '{}',      
'\x80\x02cIPython.utils.pickleutil\nCannedFunction\nq\x01)\x81q\x02}q\x03(U\x08__name__q\x04U\x05_pullq\x05U\x04codeq\x06cIPython.utils.codeutil\ncode_ctor\nq\x07(K\x01K\x02K\x05KCUOt\x00\x00|\x00\x00t\x01\x00t\x02\x00t\x03\x00f\x03\x00\x83\x02\x00r;\x00g\x00\x00|\x00\x00D]\x18\x00}\x01\x00t\x04\x00|\x01\x00t\x05\x00\x83\x00\x00\x83\x02\x00^\x02\x00q\x1f\x00St\x04\x00|\x00\x00t\x05\x00\x83\x00\x00\x83\x02\x00Sd\x01\x00Sq\x08U?helper method for implementing `client.pull` via `client.apply`q\tN\x86q\n(U\nisinstanceq\x0bU\x04listq\x0cU\x05tupleq\rU\x03setq\x0eU\x04evalq\x0fU\x07globalsq\x10tq\x11U\x04keysq\x12U\x03keyq\x13\x86q\x14U?/usr/local/lib/python2.7/dist-packages/IPython/parallel/util.pyq\x15h\x05K\xfbU\x06\x00\x03\x18\x01#\x02q\x16tRq\x17U\x07buffersq\x18]U\x06moduleq\x19U\x08__main__q\x1aU\x08defaultsq\x1bNub.', '\x80\x02}q\x01(U\x07kw_keysq\x02]q\x03U\x05nargsq\x04K\x01U\tnarg_bufsq\x05K\x01u.', '\x80\x02cIPython.utils.pickleutil\nCannedBytes\nq\x01)\x81q\x02}q\x03U\x07buffersq\x04]q\x05U\x0frandom.random()q\x06asb.']

Engine unpickles request from Hub, executes it and returns pickled results to Hub:

recv: ['038d7d3b0b83dfce7d11187a8826d434391cff6149bb25d6eaeb9bdd33e9f807', 
       '{"date":"2014-03-22T08:58:27.754746","username":"dblank","session":"d035f477-971f-4a40-88ef-3386676900ab","msg_id":"14123fd5-4406-4d5c-b9be-6fff44ed4e47","msg_type":"apply_reply"}', 
       '{"date":"2014-03-22T08:58:27.744095","username":"dblank","session":"10e5b992-2cff-4e08-af3a-b28d55848856","msg_id":"1e2fe257-63d6-40a7-8a5e-9b3b28b4ddb8","msg_type":"apply_request"}', 
       '{"dependencies_met":true,"engine":"d035f477-971f-4a40-88ef-3386676900ab","status":"ok","started":"2014-03-22T08:58:27.754341"}', 
       '{"status":"ok"}', 
       '\x80\x02G?\xe0\xaf|\xfa(E\xfd.']

Hub sends results to Noteboook Client:

send: ['kernel.12afdd67-e27f-4778-802d-bfd2312924ae.pyout', '<IDS|MSG>', '166a130936baaad24d83726d7d6f85e6c7c68fe133c657d1d1002bc83cc983b3', 
       '{"date":"2014-03-22T08:58:27.757881","username":"kernel","session":"b13bfb83-70d6-4bbe-bcfe-f51da8dbfe58","msg_id":"3d3db7a7-64de-4acf-a413-1ccf073e97ea","msg_type":"pyout"}',
       '{"username":"username","msg_id":"7331A22A3B5C4ED69D4F529A4AED457F","msg_type":"execute_request","session":"709DCA0E6DE44B51B98C39D57CBB646C"}', 
       '{}', 
       '{"execution_count":17,"data":{"text/plain":"[0.521421898449432, 0.06577071488600983]"},"metadata":{}}']