Python yield與實現(xiàn)方法代碼分析
yield的功能類似于return,但是不同之處在于它返回的是生成器。
生成器
生成器是通過一個或多個yield表達(dá)式構(gòu)成的函數(shù),每一個生成器都是一個迭代器(但是迭代器不一定是生成器)。
如果一個函數(shù)包含yield關(guān)鍵字,這個函數(shù)就會變?yōu)橐粋€生成器。
生成器并不會一次返回所有結(jié)果,而是每次遇到y(tǒng)ield關(guān)鍵字后返回相應(yīng)結(jié)果,并保留函數(shù)當(dāng)前的運行狀態(tài),等待下一次的調(diào)用。
由于生成器也是一個迭代器,那么它就應(yīng)該支持next方法來獲取下一個值。
基本操作
# 通過`yield`來創(chuàng)建生成器 def func(): for i in xrange(10); yield i
# 通過列表來創(chuàng)建生成器 [i for i in xrange(10)] # 通過`yield`來創(chuàng)建生成器 def func(): for i in xrange(10); yield i # 通過列表來創(chuàng)建生成器 [i for i in xrange(10)] Python # 調(diào)用如下 >>> f = func() >>> f # 此時生成器還沒有運行 <generator object func at 0x7fe01a853820> >>> f.next() # 當(dāng)i=0時,遇到y(tǒng)ield關(guān)鍵字,直接返回 >>> f.next() # 繼續(xù)上一次執(zhí)行的位置,進(jìn)入下一層循環(huán) ... >>> f.next() >>> f.next() # 當(dāng)執(zhí)行完最后一次循環(huán)后,結(jié)束yield語句,生成StopIteration異常 Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration >>> # 調(diào)用如下 >>> f = func() >>> f # 此時生成器還沒有運行 <generator object func at 0x7fe01a853820> >>> f.next() # 當(dāng)i=0時,遇到y(tǒng)ield關(guān)鍵字,直接返回 >>> f.next() # 繼續(xù)上一次執(zhí)行的位置,進(jìn)入下一層循環(huán) ... >>> f.next() >>> f.next() # 當(dāng)執(zhí)行完最后一次循環(huán)后,結(jié)束yield語句,生成StopIteration異常 Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration >>>
除了next函數(shù),生成器還支持send函數(shù)。該函數(shù)可以向生成器傳遞參數(shù)。
>>> def func(): ... n = 0 ... while 1: ... n = yield n #可以通過send函數(shù)向n賦值 ... >>> f = func() >>> f.next() # 默認(rèn)情況下n為0 >>> f.send(1) #n賦值1 >>> f.send(2) >>> >>> def func(): ... n = 0 ... while 1: ... n = yield n #可以通過send函數(shù)向n賦值 ... >>> f = func() >>> f.next() # 默認(rèn)情況下n為0 >>> f.send(1) #n賦值1 >>> f.send(2) >>>
應(yīng)用
最經(jīng)典的例子,生成無限序列。
常規(guī)的解決方法是,生成一個滿足要求的很大的列表,這個列表需要保存在內(nèi)存中,很明顯內(nèi)存限制了這個問題。
def get_primes(start): for element in magical_infinite_range(start): if is_prime(element): return element def get_primes(start): for element in magical_infinite_range(start): if is_prime(element): return element
如果使用生成器就不需要返回整個列表,每次都只是返回一個數(shù)據(jù),避免了內(nèi)存的限制問題。
def get_primes(number): while True: if is_prime(number): yield number number += 1 def get_primes(number): while True: if is_prime(number): yield number number += 1
生成器源碼分析
生成器的源碼在Objects/genobject.c。
調(diào)用棧
在解釋生成器之前,需要講解一下Python虛擬機(jī)的調(diào)用原理。
Python虛擬機(jī)有一個棧幀的調(diào)用棧,其中棧幀的是PyFrameObject,位于Include/frameobject.h。
typedef struct _frame { PyObject_VAR_HEAD struct _frame *f_back; /* previous frame, or NULL */ PyCodeObject *f_code; /* code segment */ PyObject *f_builtins; /* builtin symbol table (PyDictObject) */ PyObject *f_globals; /* global symbol table (PyDictObject) */ PyObject *f_locals; /* local symbol table (any mapping) */ PyObject **f_valuestack; /* points after the last local */ /* Next free slot in f_valuestack. Frame creation sets to f_valuestack. Frame evaluation usually NULLs it, but a frame that yields sets it to the current stack top. */ PyObject **f_stacktop; PyObject *f_trace; /* Trace function */ /* If an exception is raised in this frame, the next three are used to * record the exception info (if any) originally in the thread state. See * comments before set_exc_info() -- it's not obvious. * Invariant: if _type is NULL, then so are _value and _traceback. * Desired invariant: all three are NULL, or all three are non-NULL. That * one isn't currently true, but "should be". */ PyObject *f_exc_type, *f_exc_value, *f_exc_traceback; PyThreadState *f_tstate; int f_lasti; /* Last instruction if called */ /* Call PyFrame_GetLineNumber() instead of reading this field directly. As of 2.3 f_lineno is only valid when tracing is active (i.e. when f_trace is set). At other times we use PyCode_Addr2Line to calculate the line from the current bytecode index. */ int f_lineno; /* Current line number */ int f_iblock; /* index in f_blockstack */ PyTryBlock f_blockstack[CO_MAXBLOCKS]; /* for try and loop blocks */ PyObject *f_localsplus[1]; /* locals+stack, dynamically sized */ } PyFrameObject; typedef struct _frame { PyObject_VAR_HEAD struct _frame *f_back; /* previous frame, or NULL */ PyCodeObject *f_code; /* code segment */ PyObject *f_builtins; /* builtin symbol table (PyDictObject) */ PyObject *f_globals; /* global symbol table (PyDictObject) */ PyObject *f_locals; /* local symbol table (any mapping) */ PyObject **f_valuestack; /* points after the last local */ /* Next free slot in f_valuestack. Frame creation sets to f_valuestack. Frame evaluation usually NULLs it, but a frame that yields sets it to the current stack top. */ PyObject **f_stacktop; PyObject *f_trace; /* Trace function */ /* If an exception is raised in this frame, the next three are used to * record the exception info (if any) originally in the thread state. See * comments before set_exc_info() -- it's not obvious. * Invariant: if _type is NULL, then so are _value and _traceback. * Desired invariant: all three are NULL, or all three are non-NULL. That * one isn't currently true, but "should be". */ PyObject *f_exc_type, *f_exc_value, *f_exc_traceback; PyThreadState *f_tstate; int f_lasti; /* Last instruction if called */ /* Call PyFrame_GetLineNumber() instead of reading this field directly. As of 2.3 f_lineno is only valid when tracing is active (i.e. when f_trace is set). At other times we use PyCode_Addr2Line to calculate the line from the current bytecode index. */ int f_lineno; /* Current line number */ int f_iblock; /* index in f_blockstack */ PyTryBlock f_blockstack[CO_MAXBLOCKS]; /* for try and loop blocks */ PyObject *f_localsplus[1]; /* locals+stack, dynamically sized */ } PyFrameObject;
棧幀保存了給出代碼的的信息和上下文,其中包含最后執(zhí)行的指令,全局和局部命名空間,異常狀態(tài)等信息。f_valueblock保存了數(shù)據(jù),b_blockstack保存了異常和循環(huán)控制方法。
舉一個例子來說明,
def foo(): x = 1 def bar(y): z = y + 2 # def foo(): x = 1 def bar(y): z = y + 2 #
那么,相應(yīng)的調(diào)用棧如下,一個py文件,一個類,一個函數(shù)都是一個代碼塊,對應(yīng)者一個Frame,保存著上下文環(huán)境以及字節(jié)碼指令。
c --------------------------- a | bar Frame | -> block stack: [] l | (newest) | -> data stack: [1, 2] l --------------------------- | foo Frame | -> block stack: [] s | | -> data stack: [.bar at 0x10d389680>, 1] t --------------------------- a | main (module) Frame | -> block stack: [] c | (oldest) | -> data stack: [] k --------------------------- c --------------------------- a | bar Frame | -> block stack: [] l | (newest) | -> data stack: [1, 2] l --------------------------- | foo Frame | -> block stack: [] s | | -> data stack: [.bar at 0x10d389680>, 1] t --------------------------- a | main (module) Frame | -> block stack: [] c | (oldest) | -> data stack: [] k ---------------------------
每一個棧幀都擁有自己的數(shù)據(jù)棧和block棧,獨立的數(shù)據(jù)棧和block棧使得解釋器可以中斷和恢復(fù)棧幀(生成器正式利用這點)。
Python代碼首先被編譯為字節(jié)碼,再由Python虛擬機(jī)來執(zhí)行。一般來說,一條Python語句對應(yīng)著多條字節(jié)碼(由于每條字節(jié)碼對應(yīng)著一條C語句,而不是一個機(jī)器指令,所以不能按照字節(jié)碼的數(shù)量來判斷代碼性能)。
調(diào)用dis模塊可以分析字節(jié)碼,
from dis import dis dis(foo) 0 LOAD_CONST 1 (1) # 加載常量1 3 STORE_FAST 0 (x) # x賦值為1 6 LOAD_CONST 2 (<code>) # 加載常量2 9 MAKE_FUNCTION 0 # 創(chuàng)建函數(shù) 12 STORE_FAST 1 (bar) 15 LOAD_FAST 1 (bar) 18 LOAD_FAST 0 (x) 21 CALL_FUNCTION 1 # 調(diào)用函數(shù) 24 RETURN_VALUE </code> from dis import dis dis(foo) 0 LOAD_CONST 1 (1) # 加載常量1 3 STORE_FAST 0 (x) # x賦值為1 6 LOAD_CONST 2 (<code>) # 加載常量2 9 MAKE_FUNCTION 0 # 創(chuàng)建函數(shù) 12 STORE_FAST 1 (bar) 15 LOAD_FAST 1 (bar) 18 LOAD_FAST 0 (x) 21 CALL_FUNCTION 1 # 調(diào)用函數(shù) 24 RETURN_VALUE </code>
其中,
第一行為代碼行號;
第二行為偏移地址;
第三行為字節(jié)碼指令;
第四行為指令參數(shù);
第五行為參數(shù)解釋。
第一行為代碼行號;
第二行為偏移地址;
第三行為字節(jié)碼指令;
第四行為指令參數(shù);
第五行為參數(shù)解釋。
生成器源碼分析
由了上面對于調(diào)用棧的理解,就可以很容易的明白生成器的具體實現(xiàn)。
生成器的源碼位于object/genobject.c。
生成器的創(chuàng)建
PyObject * PyGen_New(PyFrameObject *f) { PyGenObject *gen = PyObject_GC_New(PyGenObject, &PyGen_Type); # 創(chuàng)建生成器對象 if (gen == NULL) { Py_DECREF(f); return NULL; } gen->gi_frame = f; # 賦予代碼塊 Py_INCREF(f->f_code); # 引用計數(shù)+1 gen->gi_code = (PyObject *)(f->f_code); gen->gi_running = 0; # 0表示為執(zhí)行,也就是生成器的初始狀態(tài) gen->gi_weakreflist = NULL; _PyObject_GC_TRACK(gen); # GC跟蹤 return (PyObject *)gen; } PyObject * PyGen_New(PyFrameObject *f) { PyGenObject *gen = PyObject_GC_New(PyGenObject, &PyGen_Type); # 創(chuàng)建生成器對象 if (gen == NULL) { Py_DECREF(f); return NULL; } gen->gi_frame = f; # 賦予代碼塊 Py_INCREF(f->f_code); # 引用計數(shù)+1 gen->gi_code = (PyObject *)(f->f_code); gen->gi_running = 0; # 0表示為執(zhí)行,也就是生成器的初始狀態(tài) gen->gi_weakreflist = NULL; _PyObject_GC_TRACK(gen); # GC跟蹤 return (PyObject *)gen; }
send與next
next與send函數(shù),如下
static PyObject * gen_iternext(PyGenObject *gen) { return gen_send_ex(gen, NULL, 0); } static PyObject * gen_send(PyGenObject *gen, PyObject *arg) { return gen_send_ex(gen, arg, 0); } static PyObject * gen_iternext(PyGenObject *gen) { return gen_send_ex(gen, NULL, 0); } static PyObject * gen_send(PyGenObject *gen, PyObject *arg) { return gen_send_ex(gen, arg, 0); }
從上面的代碼中可以看到,send和next都是調(diào)用的同一函數(shù)gen_send_ex,區(qū)別在于是否帶有參數(shù)。
static PyObject * gen_send_ex(PyGenObject *gen, PyObject *arg, int exc) { PyThreadState *tstate = PyThreadState_GET(); PyFrameObject *f = gen->gi_frame; PyObject *result; if (gen->gi_running) { # 判斷生成器是否已經(jīng)運行 PyErr_SetString(PyExc_ValueError, "generator already executing"); return NULL; } if (f==NULL || f->f_stacktop == NULL) { # 如果代碼塊為空或調(diào)用棧為空,則拋出StopIteration異常 /* Only set exception if called from send() */ if (arg && !exc) PyErr_SetNone(PyExc_StopIteration); return NULL; } if (f->f_lasti == -1) { # f_lasti=1 代表首次執(zhí)行 if (arg && arg != Py_None) { # 首次執(zhí)行不允許帶有參數(shù) PyErr_SetString(PyExc_TypeError, "can't send non-None value to a " "just-started generator"); return NULL; } } else { /* Push arg onto the frame's value stack */ result = arg ? arg : Py_None; Py_INCREF(result); # 該參數(shù)引用計數(shù)+1 *(f->f_stacktop++) = result; # 參數(shù)壓棧 } /* Generators always return to their most recent caller, not * necessarily their creator. */ f->f_tstate = tstate; Py_XINCREF(tstate->frame); assert(f->f_back == NULL); f->f_back = tstate->frame; gen->gi_running = 1; # 修改生成器執(zhí)行狀態(tài) result = PyEval_EvalFrameEx(f, exc); # 執(zhí)行字節(jié)碼 gen->gi_running = 0; # 恢復(fù)為未執(zhí)行狀態(tài) /* Don't keep the reference to f_back any longer than necessary. It * may keep a chain of frames alive or it could create a reference * cycle. */ assert(f->f_back == tstate->frame); Py_CLEAR(f->f_back); /* Clear the borrowed reference to the thread state */ f->f_tstate = NULL; /* If the generator just returned (as opposed to yielding), signal * that the generator is exhausted. */ if (result == Py_None && f->f_stacktop == NULL) { Py_DECREF(result); result = NULL; /* Set exception if not called by gen_iternext() */ if (arg) PyErr_SetNone(PyExc_StopIteration); } if (!result || f->f_stacktop == NULL) { /* generator can't be rerun, so release the frame */ Py_DECREF(f); gen->gi_frame = NULL; } return result; } static PyObject * gen_send_ex(PyGenObject *gen, PyObject *arg, int exc) { PyThreadState *tstate = PyThreadState_GET(); PyFrameObject *f = gen->gi_frame; PyObject *result; if (gen->gi_running) { # 判斷生成器是否已經(jīng)運行 PyErr_SetString(PyExc_ValueError, "generator already executing"); return NULL; } if (f==NULL || f->f_stacktop == NULL) { # 如果代碼塊為空或調(diào)用棧為空,則拋出StopIteration異常 /* Only set exception if called from send() */ if (arg && !exc) PyErr_SetNone(PyExc_StopIteration); return NULL; } if (f->f_lasti == -1) { # f_lasti=1 代表首次執(zhí)行 if (arg && arg != Py_None) { # 首次執(zhí)行不允許帶有參數(shù) PyErr_SetString(PyExc_TypeError, "can't send non-None value to a " "just-started generator"); return NULL; } } else { /* Push arg onto the frame's value stack */ result = arg ? arg : Py_None; Py_INCREF(result); # 該參數(shù)引用計數(shù)+1 *(f->f_stacktop++) = result; # 參數(shù)壓棧 } /* Generators always return to their most recent caller, not * necessarily their creator. */ f->f_tstate = tstate; Py_XINCREF(tstate->frame); assert(f->f_back == NULL); f->f_back = tstate->frame; gen->gi_running = 1; # 修改生成器執(zhí)行狀態(tài) result = PyEval_EvalFrameEx(f, exc); # 執(zhí)行字節(jié)碼 gen->gi_running = 0; # 恢復(fù)為未執(zhí)行狀態(tài) /* Don't keep the reference to f_back any longer than necessary. It * may keep a chain of frames alive or it could create a reference * cycle. */ assert(f->f_back == tstate->frame); Py_CLEAR(f->f_back); /* Clear the borrowed reference to the thread state */ f->f_tstate = NULL; /* If the generator just returned (as opposed to yielding), signal * that the generator is exhausted. */ if (result == Py_None && f->f_stacktop == NULL) { Py_DECREF(result); result = NULL; /* Set exception if not called by gen_iternext() */ if (arg) PyErr_SetNone(PyExc_StopIteration); } if (!result || f->f_stacktop == NULL) { /* generator can't be rerun, so release the frame */ Py_DECREF(f); gen->gi_frame = NULL; } return result; }
字節(jié)碼的執(zhí)行
PyEval_EvalFrameEx函數(shù)的功能為執(zhí)行字節(jié)碼并返回結(jié)果。
# 主要流程如下, for (;;) { switch(opcode) { # opcode為操作碼,對應(yīng)著各種操作 case NOP: goto fast_next_opcode; ... ... case YIELD_VALUE: # 如果操作碼是yield retval = POP(); f->f_stacktop = stack_pointer; why = WHY_YIELD; goto fast_yield; # 利用goto跳出循環(huán) } } fast_yield: ... return vetval; # 返回結(jié)果 # 主要流程如下, for (;;) { switch(opcode) { # opcode為操作碼,對應(yīng)著各種操作 case NOP: goto fast_next_opcode; ... ... case YIELD_VALUE: # 如果操作碼是yield retval = POP(); f->f_stacktop = stack_pointer; why = WHY_YIELD; goto fast_yield; # 利用goto跳出循環(huán) } } fast_yield: ... return vetval; # 返回結(jié)果
舉一個例子,f_back上一個Frame,f_lasti上一次執(zhí)行的指令的偏移量,
import sys from dis import dis def func(): f = sys._getframe(0) print f.f_lasti print f.f_back yield 1 print f.f_lasti print f.f_back yield 2 a = func() dis(func) a.next() a.next() import sys from dis import dis def func(): f = sys._getframe(0) print f.f_lasti print f.f_back yield 1 print f.f_lasti print f.f_back yield 2 a = func() dis(func) a.next() a.next()
結(jié)果如下,其中第三行的英文為操作碼,對應(yīng)著上面的opcode,每次switch都是在不同的opcode之間進(jìn)行選擇。
Python 0 LOAD_GLOBAL 0 (sys) 3 LOAD_ATTR 1 (_getframe) 6 LOAD_CONST 1 (0) 9 CALL_FUNCTION 1 12 STORE_FAST 0 (f) 15 LOAD_FAST 0 (f) 18 LOAD_ATTR 2 (f_lasti) 21 PRINT_ITEM 22 PRINT_NEWLINE 23 LOAD_FAST 0 (f) 26 LOAD_ATTR 3 (f_back) 29 PRINT_ITEM 30 PRINT_NEWLINE 31 LOAD_CONST 2 (1) 34 YIELD_VALUE # 此時操作碼為YIELD_VALUE,直接跳轉(zhuǎn)上述goto語句,此時f_lasti為當(dāng)前指令,f_back為當(dāng)前frame 35 POP_TOP 36 LOAD_FAST 0 (f) 39 LOAD_ATTR 2 (f_lasti) 42 PRINT_ITEM 43 PRINT_NEWLINE 44 LOAD_FAST 0 (f) 47 LOAD_ATTR 3 (f_back) 50 PRINT_ITEM 51 PRINT_NEWLINE 52 LOAD_CONST 3 (2) 55 YIELD_VALUE 56 POP_TOP 57 LOAD_CONST 0 (None) 60 RETURN_VALUE <frame object at 0x7fa75fcebc20> #和下面的frame相同,屬于同一個frame,也就是說在同一個函數(shù)(命名空間)內(nèi),frame是同一個。 <frame object at 0x7fa75fcebc20> 0 LOAD_GLOBAL 0 (sys) 3 LOAD_ATTR 1 (_getframe) 6 LOAD_CONST 1 (0) 9 CALL_FUNCTION 1 12 STORE_FAST 0 (f) 15 LOAD_FAST 0 (f) 18 LOAD_ATTR 2 (f_lasti) 21 PRINT_ITEM 22 PRINT_NEWLINE 23 LOAD_FAST 0 (f) 26 LOAD_ATTR 3 (f_back) 29 PRINT_ITEM 30 PRINT_NEWLINE 31 LOAD_CONST 2 (1) 34 YIELD_VALUE # 此時操作碼為YIELD_VALUE,直接跳轉(zhuǎn)上述goto語句,此時f_lasti為當(dāng)前指令,f_back為當(dāng)前frame 35 POP_TOP 36 LOAD_FAST 0 (f) 39 LOAD_ATTR 2 (f_lasti) 42 PRINT_ITEM 43 PRINT_NEWLINE 44 LOAD_FAST 0 (f) 47 LOAD_ATTR 3 (f_back) 50 PRINT_ITEM 51 PRINT_NEWLINE 52 LOAD_CONST 3 (2) 55 YIELD_VALUE 56 POP_TOP 57 LOAD_CONST 0 (None) 60 RETURN_VALUE <frame object at 0x7fa75fcebc20> #和下面的frame相同,屬于同一個frame,也就是說在同一個函數(shù)(命名空間)內(nèi),frame是同一個。 <frame object at 0x7fa75fcebc20>
總結(jié)
以上所述是小編給大家介紹的Python yield與實現(xiàn)方法代碼分析,希望對大家有所幫助,如果大家有任何疑問請給我留言,小編會及時回復(fù)大家的。在此也非常感謝大家對腳本之家網(wǎng)站的支持!
相關(guān)文章
基于Python制作一個端午節(jié)相關(guān)的小游戲
端午節(jié)快樂,今天我將為大家?guī)硪黄嘘P(guān)端午節(jié)的編程文章,希望能夠為大家獻(xiàn)上一份小小的驚喜,我們將會使用Python來實現(xiàn)一個與端午粽子相關(guān)的小應(yīng)用程序,在本文中,我將會介紹如何用Python代碼制做一個“粽子拆解器”,感興趣的小伙伴歡迎閱讀2023-06-06Mac PyCharm中的.gitignore 安裝設(shè)置教程
這篇文章主要介紹了Mac PyCharm中的.gitignore 安裝設(shè)置教程,本文通過圖文并茂的形式給大家介紹的非常詳細(xì),對大家的學(xué)習(xí)或工作具有一定的參考借鑒價值,需要的朋友可以參考下2020-04-04python利用requests庫模擬post請求時json的使用教程
這篇文章主要介紹了python利用requests庫模擬post請求時json的使用 ,本文給大家介紹的非常詳細(xì),具有一定的參考借鑒價值,需要的朋友可以參考下2018-12-12Python進(jìn)階學(xué)習(xí)之特殊方法實例詳析
一般說來,特殊的方法都被用來模仿某個行為。下面這篇文章主要給大家介紹了關(guān)于Python進(jìn)階學(xué)習(xí)之特殊方法的相關(guān)資料,文中通過示例代碼介紹的非常詳細(xì),需要的朋友可以參考借鑒,下面隨著小編來一起看看吧。2017-12-12Pytorch中關(guān)于model.eval()的作用及分析
這篇文章主要介紹了Pytorch中關(guān)于model.eval()的作用及分析,具有很好的參考價值,希望對大家有所幫助。如有錯誤或未考慮完全的地方,望不吝賜教2023-02-02Django 拼接兩個queryset 或是兩個不可以相加的對象實例
這篇文章主要介紹了Django 拼接兩個queryset 或是兩個不可以相加的對象實例,具有很好的參考價值,希望對大家有所幫助。一起跟隨小編過來看看吧2020-03-03