WO2001063438A1 - Methods and apparatus for providing bit-reversal and multicast functions utilizing dma controller - Google Patents

Methods and apparatus for providing bit-reversal and multicast functions utilizing dma controller Download PDF

Info

Publication number
WO2001063438A1
WO2001063438A1 PCT/US2001/005714 US0105714W WO0163438A1 WO 2001063438 A1 WO2001063438 A1 WO 2001063438A1 US 0105714 W US0105714 W US 0105714W WO 0163438 A1 WO0163438 A1 WO 0163438A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
transfer
controller
dma
bus
Prior art date
Application number
PCT/US2001/005714
Other languages
French (fr)
Inventor
Edwin F. Barry
Nikos Pitsianis
Kevin Coopman
Original Assignee
Bops Incorporated
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Bops Incorporated filed Critical Bops Incorporated
Publication of WO2001063438A1 publication Critical patent/WO2001063438A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/14—Handling requests for interconnection or transfer
    • G06F13/20—Handling requests for interconnection or transfer for access to input/output bus
    • G06F13/28—Handling requests for interconnection or transfer for access to input/output bus using burst mode transfer, e.g. direct memory access DMA, cycle steal

Definitions

  • the present invention relates generally to improvements in array processing, and more particularly to advantageous techniques for providing improved methods and apparatus for data distribution to and collection from multiple memories often associated with and local to processing elements within an array processor.
  • Background of the Invention Various prior art techniques exist for the transfer of data between system memories or between system memories and input output (I/O) devices. Fig.
  • FIG. 1 shows a conventional data processing system 100 comprising a processor local memory 110, a host uniprocessor 120, I/O devices 130 and 140, system memory 150 which is usually a larger memory store with longer access delay than the processor local memory, and a direct memory access (DMA) controller 160.
  • DMA direct memory access
  • the DMA controller 160 provides a mechanism for transferring data between processor local memory and system memory or I/O devices concurrent with uniprocessor execution.
  • DMA controllers are sometimes referred to as I/O processors or transfer processors in the literature. System performance is improved since the host uniprocessor can perform computations while the DMA controller is transferring new input data to the processor local memory and transferring result data to output devices or the system memory.
  • a data transfer between a source and a destination is typically specified with the following minimum set of parameters: source address, destination address, and number of data elements to transfer. Addresses are interpreted by the system hardware and uniquely specify I/O devices or memory locations from which data must be read or to which data must be written. Sometimes additional parameters are provided such as data element size.
  • One of the limitations of conventional DMA controllers is that address generation capabilities for the data source and data destination are often constrained to be the same. For example, when only a source address, destination address and a transfer count are specified, the implied data access pattern is block-oriented, that is, a sequence of data words from contiguous addresses starting with the source address is copied to a sequence of contiguous addresses starting at the destination address.
  • Array processing presents challenges for data transfer both in terms of addressing flexibility, control and performance.
  • the patterns in which data elements are distributed and collected from PE local memories can significantly affect the overall performance of the processing system.
  • FFT fast Fourier transform
  • Manifold array Manifold array
  • Fig. 1 shows a conventional data processing system with a DMA controller to support data transfers concurrent with host processor computation
  • Fig. 2 shows a ManArray DSP with DMA controller in a representative system suitable for use in conjunction with the present invention
  • Fig. 3 shows a DMA controller implemented as a multiprocessor, with two transfer controllers, bus connections to a system memory, PE memories and a control bus;
  • Fig. 4 shows a single transfer controller comprising four primary execution units, bus connections and FIFOs
  • Fig. 5 shows an exemplary format of a transfer type instruction
  • FIG. 6 shows an example of virtual to physical PE ID translation
  • Fig. 7 shows an exemplary logical implementation of a system for virtual PE ID (VID) to physical PE ID (PID) translation
  • Fig. 8 shows an exemplary PE translation (PEXLAT) instruction format ("load VID- to-PID table");
  • Fig. 9 shows an exemplary VID-to-PID translation table register, called the PETABLE register in a presently preferred embodiment;
  • Fig. 10 shows illustrative bit reversal (BitRev) code suitable for carrying out an address transformation for several FFT sizes and processing methods in accordance with the present invention
  • Fig. 11 shows an exemplary encoding table for the BitRev code of Fig. 10; and Fig. 12 shows a further exemplary encoding table including PE and address bits for the BitRev code of Fig. 10.
  • Fig. 10 shows an exemplary encoding table for the BitRev code of Fig. 10; and
  • Fig. 12 shows a further exemplary encoding table including PE and address bits for the BitRev code of Fig. 10.
  • 09/596,103 entitled “Methods and Apparatus for Data Dependent Address Operations and Efficient Variable Length Code Decoding in a VLIW Processor” filed June 16, 2000
  • U.S. Patent Application Serial No. 09/598,567 entitled “Methods and Apparatus for Improved Efficiency in Pipeline Simulation and Emulation” filed June 21, 2000
  • U.S. Patent Application Serial No. 09/598,564 entitled “Methods and Apparatus for Initiating and Resynchronizing Multi-Cycle SIMD Instructions” filed June 21, 2000
  • 60/139,946 entitled “Methods and Apparatus for Data Dependent Address Operations and Efficient Variable Length Code Decoding in a VLIW Processor” filed June 18, 1999
  • Provisional Application Serial No. 60/140,245 entitled “Methods and Apparatus for Generalized Event Detection and Action Specification in a Processor” filed June 21, 1999
  • Provisional Application Serial No. 60/140,163 entitled “Methods and Apparatus for Improved Efficiency in Pipeline Simulation and Emulation” filed June 21, 1999
  • 60/140,162 entitled “Methods and Apparatus for Initiating and Re- Synchronizing Multi-Cycle SIMD Instructions” filed June 21, 1999
  • Provisional Application Serial No. 60/140,244 entitled “Methods and Apparatus for Providing One-By-One Manifold Array (lxl ManArray) Program Context Control” filed June 21, 1999
  • Provisional Application Serial No. 60/140,325 entitled “Methods and Apparatus for Establishing Port Priority Function in a VLIW Processor” filed June 21, 1999
  • 60/140,425 entitled “Methods and Apparatus for Parallel Processing Utilizing a Manifold Array (ManArray) Architecture and Instruction Syntax” filed June 22, 1999
  • Provisional Application Serial No. 60/165,337 entitled “Efficient Cosine Transform Implementations on the ManArray Architecture” filed November 12, 1999
  • Provisional Application Serial No. 60/171,911 entitled “Methods and Apparatus for DMA Loading of Very Long Instruction Word Memory” filed December 23, 1999
  • Provisional Application Serial No. 60/184,668 entitled “Methods and Apparatus for Providing Bit-Reversal and Multicast Functions Utilizing DMA Controller” filed February 24, 2000
  • Provisional Application Serial No. 60/184,529 entitled “Methods and Apparatus for Scalable Array Processor Interrupt Detection and Response” filed February 24, 2000
  • a “transfer” refers to the movement of one or more units of data from a source device (either I/O or memory) to a destination device (I/O or memory).
  • a data "source” or “destination” refers to a device from which data may be read or to which data may be written. Such a device provides a range of one or more contiguous addresses for reading and/or writing each of which corresponds to at least one data storage element. For some data sources and destinations, there may be many addresses which map to the same data storage location, or many storage locations that map to the same address. For example, an I/O device may be accessed using one of many addresses in a range of addresses, yet it will perform the same operation, such as returning the next data element of a FIFO queue, for any of them.
  • a "data access pattern” is a sequence of data source or destination addresses whose relationship to each other is periodic. For example, the sequence of addresses 0, 1, 2, 4, 5, 6, 8, 9, 10,...etc. is a data access pattern. If we look at the differences between successive addresses, we find: 1,1,2, 1,1,2, 1,1,2, ...etc. Every three elements, the pattern repeats.
  • AGU address generation unit
  • EOT End-of-transfer
  • host processor is any processor or device which can write control commands and read status from the DMA controller and/or which can respond to DMA controller messages and signals.
  • a host processor interacts with the DMA controller to control and synchronize the flow of data between devices and memories in the system in such a way as to avoid overrun and underrun conditions at the sources and destinations of data transfers.
  • the present invention provides a set of flexible addressing modes for supporting efficient data transfers to and from multiple memories, together with mechanisms for allowing data accesses to be directed to PEs according to virtual as opposed to physical IDs.
  • This section describes an exemplary DMA controller and its system environment that provides one context in which the present invention may be effectively used.
  • the discussion below addresses PE memory addressing, virtual-to-physical PE ID translation and its purpose, and a set of PE memory addressing modes or "PE addressing modes" which support numerous parallel algorithms and processes with highly efficient data transfer.
  • Fig. 2 shows an exemplary system 200 which illustrates the context in which a ManArray DMA controller 201, in accordance with the present invention, resides.
  • the DMA controller 201 accesses processor local memories 210, 211, 212, 213, 214 and 215 via the DMA bus 202, 202 h 202 2 , 202 3 , 202 4 , 202 5 and the memory interface units 205, 206, 207, 208 and 209 to which it is connected.
  • a ManArray DSP 203 also connects to its local memories 210-215 via memory interface units 205-209. Further details of a presently preferred DSP 203 are found in the above incorporated by reference applications. Although a ManArray DSP 203 is used in the preferred embodiment, the invention described herein may be used with any other processor which is coupled to an array or set of local memories such that the DMA controller has similar access capabilities.
  • the DMA controller 201 also connects to two system busses, a system control bus (SCB) 235, and the System Data Bus (SDB) 240.
  • the DMA controller 201 is designed to transfer data between devices on the SDB 240, such as the system memory 250 and the DSP 203 local memories 210-215.
  • the SCB 235 is used by an SCB master, such as the DSP 203 or a host control processor (HCP) 245, to access various system and DMA control registers.
  • the DMA control registers are provided for initiating transfer or semaphore control operations and for reading transfer controller status.
  • the SCB 235 is also used by the DMA controller 201 to send synchronization messages to other SCB bus slaves such as DSP control registers 225 and host I/O block 255. Some of the DSP control registers 225 act as "mailboxes" for receiving messages which may be sent by a DMA transfer controller or by another SCB master such as the HCP.
  • Fig. 3 shows a system 300 which illustrates the operation of a DMA controller 301 which may suitably be a multiprocessor specialized to carry out data transfers utilizing one or more transfer controllers 302 and 303.
  • Each transfer controller can operate as an independent processor or work together with other transfer controllers to carry out data transfers.
  • DMA busses 305, 310 provide, in the presently preferred embodiment, independent data paths to processor local memories 320, 321, 322, 323, 324, 325, one bus "lane” consisting of address, control, and data signals for each of the transfer controllers 302 and 303.
  • each transfer controller is connected to an SDB 350 and to an SCB 330.
  • Each transfer controller operates as a bus master and a bus slave on both the SCB and SDB.
  • a transfer controller may be accessed by other SCB bus masters in order to read its internal state or issue control commands.
  • a transfer controller can send synchronization messages to other SCB bus slaves.
  • a transfer controller performs data reads and writes from or to system memory or I/O devices which are bus slaves on the SDB.
  • a transfer controller can cooperate with another SDB bus master in a "slave mode" allowing the bus master to read or write data directly from or to its data FIFOs (as discussed further below).
  • the DMA busses 305 and 310, the SDB 350, and the SCB 330 may be implemented in different ways. For example, they may be implemented with varying bus widths, protocols, or the like consistent with the teachings of the present invention.
  • Fig. 4 shows a system 400 having a single transfer controller 401 comprising a set of execution units including an instruction control unit (ICU) 440, a system transfer unit (STU) 402, a core transfer unit (CTU) 408 and an event control unit (ECU) 460.
  • ICU instruction control unit
  • STU system transfer unit
  • CTU core transfer unit
  • ECU event control unit
  • core and “local” in the text are used as adjectives to distinguish resources or components which are primarily used by and directly connected to the "core” processor versus those which are primarily used by or connected to a host control processor.
  • the term "system” is used as an adjective when referring to resources used primarily by the host control processor.
  • An inbound data queue (IDQ) 405 is a data FIFO buffer which is written with data from an SDB 470 under control of the STU 402.
  • Data is read from the IDQ 405 under control of the CTU 408 to be sent to local memories 430, or sent to the ICU 440 in the case of instruction fetches.
  • An outbound data queue (ODQ) 406 is a data FIFO which is written with data from DMA busses 425 under control of the CTU 408, to be sent to an SDB 470 device or memory under the control of the STU 402.
  • the CTU 408 may also read DMA instructions from any one of the local memories attached to the DMA busses 425. These DMA instructions are then forwarded to the ICU 440 for initial decoding.
  • the ECU 460 receives signal inputs from external devices 465, commands from the SCB 450, and instruction data from the ICU 440.
  • Each transfer controller within a ManArray DMA controller is designed to fetch its own stream of DMA instructions.
  • DMA instructions are of five basic types: transfer; branch; load; synchronization; and state control.
  • the branch, load, synchronization and state control types of instructions are collectively referred to as "control instructions", and distinguished from the transfer instructions which actually perform data transfers.
  • DMA instructions in the exemplary system are typically of multi-word length and require a variable number of cycles to execute in order to minimize DMA instruction storage requirements.
  • transfer-type instructions are dispatched by an ICU 440 for further decoding and execution by an STU 402 and a CTU 408.
  • a "transfer- system-inbound” or TSI instruction moves data from SDB 470 to IDQ 405 and is executed by the STU 402.
  • a "transfer-core-inbound” (TCI) instruction moves data from the IDQ 405 to the DMA Bus 425 and is executed by the CTU.
  • TCO transfer-core-outbound
  • TSO transfer-system-outbound
  • Two transfer instructions are required to move data between an SDB system memory and one or more SP or PE local memories on the DMA bus, and both instructions are executed concurrently: a TSI, TCI pair or a TSO, TCO pair.
  • the address parameter of the STU transfer instructions, TSI and TSO refers to addresses on the SDB while the address parameter of the CTU transfer instructions, TCI and TCO, refers to addresses on the DMA bus which target local memories.
  • Fig. 5 shows an exemplary instruction format 500 for transfer instructions.
  • a base opcode field 501 indicates that the instruction is of transfer type.
  • a C/S bit 510 indicates the transfer unit (CTU or STU) and I/O bit 520 indicates whether the transfer direction is inbound or outbound.
  • Execute (“X") bit 550 when set to 1, indicates that the transfer should start immediately after decoding the transfer instruction. The execute bit provides one means of specifying a "start transfer" event. When the X bit is 0, the transfer parameters are decoded and loaded into the specified transfer unit, either the STU or the CTU, but the transfer is not initiated. Instruction fetch/decode continues normally until a "start transfer" event occurs.
  • Data type field 530 indicates the size of each element transferred and address mode 540 refers to the data access pattern which must be generated by the transfer unit.
  • a transfer count 560 indicates the number of data elements of size "data type" which are to be transferred to or from the target memory/device before an EOT occurs for that unit.
  • An address parameter 570 specifies the starting address for the transfer. Other parameters 580 may follow the address word of the instruction, depending on the addressing mode used.
  • the PE address modes access only the set of PE memories 210, 211, 212, and 213 in this exemplary ManArray DSP configuration.
  • the address of a data element within PE local memory space is specified with three variables, a PE ID, a base value, and an index value.
  • the base and the index values are summed to form a virtual offset into a PE memory relative to an address 0, the first address of that PE's memory.
  • This virtual offset is then translated, via an address permutation and selection mechanism into a physical offset.
  • the permutation and selection mechanism, together with PE ID translation, as described below, is designed to support data reorderings required for computations, such as FFT computations.
  • the address of a PE data element may be thought of as a function of both the PE ID and the permuted sum of a base value and an index value:
  • PE data address (PE ID, BitReversalSelect(Base + Index)), where the function "BitReversalSelect” is the permutation and selection function required to support FFT data reorderings within each local PE memory.
  • the ManArray architecture incorporates a unique interconnection network between processing elements (PEs) which uses PE virtual IDs (VIDs) to support useful single-cycle communication paths, for example, torus and hypercube paths. In some array organizations, the PE's physical and virtual IDs are equal.
  • the VIDs are used in the ManArray architecture to specify the pattern for data distribution and collection. When data is distributed according to the pattern established by VID assignment, then efficient inter-PE communication required by the programmer becomes available. To this end, Fig.
  • FIG. 6 shows an exemplary table 600 illustrating one particular VID-to-PID translation.
  • Fig. 7 shows an exemplary logical implementation of a system 700 for VID-to-PID translation.
  • Fig. 8 shows an exemplary PExLAT instruction 800.
  • Fig. 9 shows an exemplary VID-to-PID translation table register 900.
  • Such aspects of a presently preferred embodiment are described in further detail in U.S. Patent Application Serial No. 09/472,372 entitled "Methods and Apparatus for Providing Direct Memory Access Control" and filed December 23, 1999.
  • the data will be distributed to the PEs according to a VID assignment in such a manner that the physical switch connections allow data to be transferred between PEs as though the switch topology were a hypercube even if the switch connections between physical PEs do not support the full hyper-cube interconnectivity.
  • the present invention describes two approaches whereby the DMA controller can access PE memories according to their VIDs, effectively mapping PE virtual IDs to PE physical IDs (PIDs).
  • the first uses VID-to-PID translation within the CTU of a transfer controller. This translation can be performed either through table-lookup, or through logic permutations on the VID.
  • the second approach associates a VID with a PE by providing a programmable register within the PE or a PE local memory interface unit (LMIU) such as LMIUs 205, 206, 207 and 208 of Fig. 2, which is used by the LMIU logic to "capture" a data access when its VID matches a VID provided on the DMA Bus for each DMA memory access.
  • LMIU PE local memory interface unit
  • Bit-reverse PE addressing allows efficient scatter and gather of FFT data and coefficients.
  • the DMA controller provides an efficient means for post-processing FFT calculations through its bit-reverse addressing capability.
  • Bit reversal is a transposition of bits where the most significant bit (of a given "field” width) becomes least significant, and so on. For example, 0001011 will become 0011010 when the field width is 5 bits.
  • Digit reversal is a transposition of groups of bits (a group of bits defines a digit) where the most significant digit becomes least significant and vice versa. For example, 000101 1 will become 0111000 for field width 6 and digit width 2.
  • Algorithms with radix larger than 8 seem to be impractical for a number of reasons, so typically only digit widths 1-3 are encountered. Also, any algorithm is capable in processing its own output, meaning that if algorithm A takes in-order input and produces some of the above reversals, then for the inverse transform, algorithm A with a reversed input of the same kind, will produce an in-order output. Complications arise when one is faced with providing or dealing with a reversed ordering that is not the result of the algorithm at hand.
  • an algorithm is required to: 1) take a distributed reversed output from the PEs and generate an in-order vector in external memory, or 2) take a distributed reversed output from the PEs and generate a bit-reversed vector, or 3) provide the inverse of either of the above two.
  • the offset or vector index can be considered to consist of two fields: a) the distributed address (usually this is the PE id) consisting of the most significant bits, and b) the local address (rest of the bits). To achieve 1) above, bit-reverse PE IDs and then digit reverse local address according to radix.
  • a "bit reversal code" is stored in a DMA register, BITREV, whose default value is 0 specifying no reversal. This register is read/writeable from the MCB. There is one BITREV register per transfer controller and this register is reset to 0 on DMA RESET.
  • a presently preferred BITREV register encoding format 1000 is shown in Fig. 10. The exemplary system supports bit-reverse addressing for radix 2, 4, and 8 FFTs, which corresponds to digit widths of 1,2, and 3 bits respectively.
  • the system also supports reversed orderings (1) and (2) above and FFT sizes 256, 512, 1024, 2048, 4096 and 8192 that use 6, 7, 8, 9, 10, and 11 bits per PE address, respectively. These bit requirements are defined in the exemplary system to support 32-bit word addressing on the DMA busses so that the least significant 2 bits are assumed to be zero.
  • PEXLAT must also be loaded to achieve the specified bit-reversed address operation.
  • PEXLAT is a register that may be loaded either by MCB write access or by a DMA instruction. It specifies the order in which PEs are accessed for PE addressing modes. Note for Cooley-Tukey FFTs PEXLAT for inbound transfers is ⁇ 0,1,2,3 ⁇ and for outbound transfers the ordering is ⁇ 0,2,1,3 ⁇ . Details of an exemplary PEXLAT instruction 800 are shown in Fig. 8.
  • PE address bits [1:0] are always assumed to be zero since they are not routed to the PEs, since DMA transfers, in the present exemplary system, are only in 32-bit words.
  • the PE address bits for each valid BitRev code encoding with their corresponding (triple) are shown in table 1200 of Fig. 12.
  • a PE operation code refers to a set of signals and cycle types on the DMA bus which specify operations to be performed at the local memory interface unit (LMIU), or within the PEs themselves.
  • this signal group is used to specify operations such as:
  • a specific PE which is to send/receive data is to send/receive data
  • a group of 2 or more PEs which are to receive data is to be received
  • a group of PEs which are to send data on specific data lines of the DMA Bus (each PE uses a set of wires depending on its PE ID using virtual or physical IDs, depending on implementation,
  • a group of PEs which are to read data from specific data lines of the DMA Bus (each PE uses a set of wires depending on its PE ID (virtual or physical, depending on implementation),
  • DMA virtual channel ID used to control the update of semaphores associated with local memory access by PEs or DMA lane, or
  • Load PE Opcode - a bit that says that corresponding data is a PE Opcode
  • Enable PE Opcode - a bit that enables the use of the PE Opcode in LMIU on current data
  • Disable PE Opcode - a bit that disables use of the PE opcode in LMIU on current data
  • Data Operation - a bit that specifies that the access is a data operation using current setting of PE Opcode (if not disabled), or
  • PE ID - indicates a specific PE address for normal data accesses (no PE Opcode used).
  • the data bus can be used to specify a PE opcode.
  • One of the bits in the set of signals indicates that the cycle type is "Load new PE Opcode".
  • the PE opcode state is disabled when the last cycle of a transfer is presented to the LMIU.
  • PE multicast addressing refers to parallel distribution of the same data element to more than one, and up to all, PEs simultaneously. In other words, for the all PE case it refers to a "broadcast" operation.
  • Multicast addressing is specified as a set of addressing modes in the TCI instruction, as in the address mode field 540 of Fig. 5.
  • another parameter in the transfer instruction specifies which PEs are to accept the data transfer.
  • a 16 bit field can support the specification of any combination of up to 16 PEs to receive the same data element.
  • a 4 bit number can specify any set of 16 PEs to receive the same data element when used in combination with the PE VID-to-PID translation table.
  • An encoded M-bit value can be used to specify selected, common combinations of
  • PEs to receive data simultaneously.
  • multicast addressing can be used along with block, stride and circular addressing to any group of PEs.
  • the PE relative packing-gather operations type indicates that each PE drives data onto a different group of data wires to return to the transfer controller ODQ. For example, a byte size gather over 4 PEs would specify that each PE supplies one byte of a 32-bit word to be returned to the ODQ for transfer to the system data bus. Other data types may be supported, with varying numbers of PEs participating in the gather operation.
  • a specified number of data elements are summed as they are read from local memories. If the summing parameter is N, then a single data element is transferred to the ODQ for every N elements read from local memories. The N elements are summed with the result being placed in the ODQ for transfer to the system data bus.
  • the PE Relative Unpack-Distribute Operations type indicates that each PE LMIU receives data from a different group of data wires to be written to its local memory. For example, a byte size unpack-distribute over 4 PEs would specify that each PE reads one byte of a 32-bit word to be subsequently written to the local memory.
  • Optional zero-extend, sign-extend or one- extend may be supported.
  • Other data types may be supported, such as halfwords, with varying numbers of PEs participating in the gather operation.
  • the DMA Bus width influences the number of useful data types which might be supported.
  • data may be permuted before being sent to PEs for inbound transfers, or before being sent to system memories for outbound transfers. Besides reordering of data within a data element, other stream oriented operations may be performed including masking, data merging such as AND with a mask followed by OR with a constant, complementation using XOR with a specified mask.

Abstract

Techniques for providing improved data distribution to and collection from multiple memories are described. Such memories are (210, 211, 212, 213, 214, 215) often associated with and local to processing elements (PEs) within an array processor (200). Improved data transfer control within a data processing system provides support for radix 2, 4 and 8 fast Fourier transform (FFT) algorithms through data reordering or bit-reversed addressing across multiple PEs, carried out concurently with FFT computation on a digital signal processor (DSP) array (203) by a DMA unit (201). Parallel data distribution and collection through forms of multicast and packet-gather operations are also supported.

Description

METHODS AND APPARATUS FOR PROVIDING BIT-REVERSAL AND MULTICAST FUNCTIONS UTILIZING DMA CONTROLLER
The present invention claims the benefit of U.S. Provisional Application Serial No.
60/184,668 entitled "Methods and Apparatus for Providing Bit-Reversal and Multicast Functions Utilizing DMA Controller" filed February 24, 2000 and incorporated by reference herein in its entirety. Field of the Invention The present invention relates generally to improvements in array processing, and more particularly to advantageous techniques for providing improved methods and apparatus for data distribution to and collection from multiple memories often associated with and local to processing elements within an array processor. Background of the Invention Various prior art techniques exist for the transfer of data between system memories or between system memories and input output (I/O) devices. Fig. 1 shows a conventional data processing system 100 comprising a processor local memory 110, a host uniprocessor 120, I/O devices 130 and 140, system memory 150 which is usually a larger memory store with longer access delay than the processor local memory, and a direct memory access (DMA) controller 160.
The DMA controller 160 provides a mechanism for transferring data between processor local memory and system memory or I/O devices concurrent with uniprocessor execution. DMA controllers are sometimes referred to as I/O processors or transfer processors in the literature. System performance is improved since the host uniprocessor can perform computations while the DMA controller is transferring new input data to the processor local memory and transferring result data to output devices or the system memory. A data transfer between a source and a destination is typically specified with the following minimum set of parameters: source address, destination address, and number of data elements to transfer. Addresses are interpreted by the system hardware and uniquely specify I/O devices or memory locations from which data must be read or to which data must be written. Sometimes additional parameters are provided such as data element size. One of the limitations of conventional DMA controllers is that address generation capabilities for the data source and data destination are often constrained to be the same. For example, when only a source address, destination address and a transfer count are specified, the implied data access pattern is block-oriented, that is, a sequence of data words from contiguous addresses starting with the source address is copied to a sequence of contiguous addresses starting at the destination address. Array processing presents challenges for data transfer both in terms of addressing flexibility, control and performance. The patterns in which data elements are distributed and collected from PE local memories can significantly affect the overall performance of the processing system. One important application is fast Fourier transform (FFT) processing which uses bit-reversed addressing to reorder the data elements. With the advent of the manifold array (ManArray) architecture, it has been recognized that it will be advantageous to have improved techniques for data transfer which efficiently provide these and other capabilities and which are tailored to this new architecture. Summary of the Invention As described in greater detail below, the present invention addresses a variety of advantageous approaches for improved data transfer control within a data processing system. In particular, improved techniques are provided for:
(1) Supporting radix 2, 4 and 8 fast Fourier transform algorithms through efficient data reordering or "bit-reversed addressing" across multiple processing elements (PEs), carried out concurrently with FFT computation by a digital signal processor (DSP), and
(2) Parallel data distribution and collection through efficient forms of multicast and "packing-gather" operations.
These and other aspects and advantages of the present invention will be apparent from the drawings and the Detailed Description which follow. Brief Description of the Drawings
Fig. 1 shows a conventional data processing system with a DMA controller to support data transfers concurrent with host processor computation;
Fig. 2 shows a ManArray DSP with DMA controller in a representative system suitable for use in conjunction with the present invention;
Fig. 3 shows a DMA controller implemented as a multiprocessor, with two transfer controllers, bus connections to a system memory, PE memories and a control bus;
Fig. 4 shows a single transfer controller comprising four primary execution units, bus connections and FIFOs; Fig. 5 shows an exemplary format of a transfer type instruction;
Fig. 6 shows an example of virtual to physical PE ID translation; Fig. 7 shows an exemplary logical implementation of a system for virtual PE ID (VID) to physical PE ID (PID) translation;
Fig. 8 shows an exemplary PE translation (PEXLAT) instruction format ("load VID- to-PID table"); Fig. 9 shows an exemplary VID-to-PID translation table register, called the PETABLE register in a presently preferred embodiment;
Fig. 10 shows illustrative bit reversal (BitRev) code suitable for carrying out an address transformation for several FFT sizes and processing methods in accordance with the present invention;
Fig. 11 shows an exemplary encoding table for the BitRev code of Fig. 10; and Fig. 12 shows a further exemplary encoding table including PE and address bits for the BitRev code of Fig. 10. Detailed Description Further details of a presently preferred ManArray DSP core, architecture, and instructions for use in conjunction with the present invention are found in U.S. Patent Application Serial No. 08/885,310 filed June 30, 1997, now U.S. Patent No. 6,023,753, U.S. Patent Application Serial No. 08/949,122 filed October 10, 1997, U.S. Patent Application Serial No. 09/169,255 filed October 9, 1998, U.S. Patent Application Serial No. 09/169,256 filed October 9, 1998, U.S. Patent Application Serial No. 09/169,072 filed October 9, 1998, U.S. Patent Application Serial No. 09/187,539 filed November 6, 1998, U.S. Patent Application Serial No. 09/205,558 filed December 4, 1998, U.S. Patent Application Serial No. 09/215,081 filed December 18, 1998, U.S. Patent Application Serial No. 09/228,374 filed January 12, 1999 and entitled "Methods and Apparatus to Dynamically Reconfigure the Instruction Pipeline of an Indirect Very Long Instruction Word Scalable Processor", U.S. Patent Application Serial No. 09/238,446 filed January 28, 1999, U.S. Patent Application Serial No. 09/267,570 filed March 12, 1999, U.S. Patent Application Serial No. 09/337,839 filed June 22, 1999, U.S. Patent Application Serial No. 09/350,191 filed July 9, 1999, U.S. Patent Application Serial No. 09/422,015 filed October 21, 1999 entitled "Methods and Apparatus for Abbreviated Instruction and Configurable Processor Architecture", U.S. Patent Application Serial No. 09/432,705 filed November 2, 1999 entitled "Methods and Apparatus for Improved Motion Estimation for Video Encoding", U.S. Patent Application Serial No. 09/471,217 filed December 23, 1999 entitled "Methods and Apparatus for Providing Data Transfer Control", U.S. Patent Application Serial No. 09/472,372 filed December 23, 1999 entitled "Methods and Apparatus for Providing Direct Memory Access Control", U.S. Patent Application Serial No. 09/596,103 entitled "Methods and Apparatus for Data Dependent Address Operations and Efficient Variable Length Code Decoding in a VLIW Processor" filed June 16, 2000, U.S. Patent Application Serial No. 09/598,567 entitled "Methods and Apparatus for Improved Efficiency in Pipeline Simulation and Emulation" filed June 21, 2000, U.S. Patent Application Serial No. 09/598,564 entitled "Methods and Apparatus for Initiating and Resynchronizing Multi-Cycle SIMD Instructions" filed June 21, 2000, U.S. Patent Application Serial No. 09/598,566 entitled "Methods and Apparatus for Generalized Event Detection and Action Specification in a Processor" filed June 21, 2000, and U.S. Patent Application Serial No. 09/598,084 entitled "Methods and Apparatus for Establishing Port Priority Functions in a VLIW Processor" filed June 21, 2000, U.S. Patent Application Serial No. 09/599,980 entitled "Methods and Apparatus for Parallel Processing Utilizing a Manifold Array (ManArray) Architecture and Instruction Syntax" filed June 22, 2000, U.S. Patent
Application Serial No. entitled "Methods and Apparatus for Scalable Array Processor
Interrupt Detection and Response" filed February 23, 2001, U.S. Patent Application Serial No. entitled "Methods and Apparatus for Dual-Use Coprocessing/Debug Interface" filed February 23, 2001, as well as, Provisional Application Serial No. 60/113,637 entitled "Methods and Apparatus for Providing Direct Memory Access (DMA) Engine" filed December 23, 1998, Provisional Application Serial No. 60/113,555 entitled "Methods and Apparatus Providing Transfer Control" filed December 23, 1998, Provisional Application Serial No. 60/139,946 entitled "Methods and Apparatus for Data Dependent Address Operations and Efficient Variable Length Code Decoding in a VLIW Processor" filed June 18, 1999, Provisional Application Serial No. 60/140,245 entitled "Methods and Apparatus for Generalized Event Detection and Action Specification in a Processor" filed June 21, 1999, Provisional Application Serial No. 60/140,163 entitled "Methods and Apparatus for Improved Efficiency in Pipeline Simulation and Emulation" filed June 21, 1999, Provisional Application Serial No. 60/140,162 entitled "Methods and Apparatus for Initiating and Re- Synchronizing Multi-Cycle SIMD Instructions" filed June 21, 1999, Provisional Application Serial No. 60/140,244 entitled "Methods and Apparatus for Providing One-By-One Manifold Array (lxl ManArray) Program Context Control" filed June 21, 1999, Provisional Application Serial No. 60/140,325 entitled "Methods and Apparatus for Establishing Port Priority Function in a VLIW Processor" filed June 21, 1999, Provisional Application Serial No. 60/140,425 entitled "Methods and Apparatus for Parallel Processing Utilizing a Manifold Array (ManArray) Architecture and Instruction Syntax" filed June 22, 1999, Provisional Application Serial No. 60/165,337 entitled "Efficient Cosine Transform Implementations on the ManArray Architecture" filed November 12, 1999, and Provisional Application Serial No. 60/171,911 entitled "Methods and Apparatus for DMA Loading of Very Long Instruction Word Memory" filed December 23, 1999, Provisional Application Serial No. 60/184,668 entitled "Methods and Apparatus for Providing Bit-Reversal and Multicast Functions Utilizing DMA Controller" filed February 24, 2000, Provisional Application Serial No. 60/184,529 entitled "Methods and Apparatus for Scalable Array Processor Interrupt Detection and Response" filed February 24, 2000, Provisional Application Serial No.
60/184,560 entitled "Methods and Apparatus for Flexible Strength Coprocessing Interface" filed February 24, 2000, Provisional Application Serial No. 60/203,629 entitled "Methods and Apparatus for Power Control in a Scalable Array of Processor Elements" filed May 12, 2000, Provisional Application Serial No. 60/241,940 entitled "Methods and Apparatus for Efficient Vocoder Implementations" filed October 20, 2000, and Provisional Application Serial No. 60/251,072 entitled "Methods and Apparatus for Providing Improved Physical Designs and Routing with Reduced Capacitive Power Dissipation" filed December 4, 2000, all of which are assigned to the assignee of the present invention and incorporated by reference herein in their entirety.
The following definitions of terms are provided as background for the discussion of the invention which follows below:
A "transfer" refers to the movement of one or more units of data from a source device (either I/O or memory) to a destination device (I/O or memory).
A data "source" or "destination" refers to a device from which data may be read or to which data may be written. Such a device provides a range of one or more contiguous addresses for reading and/or writing each of which corresponds to at least one data storage element. For some data sources and destinations, there may be many addresses which map to the same data storage location, or many storage locations that map to the same address. For example, an I/O device may be accessed using one of many addresses in a range of addresses, yet it will perform the same operation, such as returning the next data element of a FIFO queue, for any of them.
A "data access pattern" is a sequence of data source or destination addresses whose relationship to each other is periodic. For example, the sequence of addresses 0, 1, 2, 4, 5, 6, 8, 9, 10,...etc. is a data access pattern. If we look at the differences between successive addresses, we find: 1,1,2, 1,1,2, 1,1,2, ...etc. Every three elements, the pattern repeats. An "address mode" or "addressing mode" refers to a rule that describes a sequence of addresses, usually in terms of one or more parameters. For example, a "block" address mode is described by the rule: address[i] = base address + i, where i = 0, 1 , 2, ...etc. and where base address is a parameter and refers to the starting address of the sequence. Another example is a "stride" address mode which may be described by the rule: address[i] = base_address + (i mod (stride - hold)) + (i / hold) * stride for i = 0, 1, 2, ...etc., and where base_address, stride and hold are parameters, and where division is integer division such that any remainder is discarded.
An "address generation unit" (AGU) is a hardware module that generates a sequence of addresses (a data access pattern) according to a programmed address mode. "End-of-transfer" (EOT) refers to the state when a transfer execution unit, as further described in the following text, has completed its most recent transfer instruction by transferring the last of the number of elements specified by the instruction's transfer count field.
The term "host processor" as used in the following descriptions is any processor or device which can write control commands and read status from the DMA controller and/or which can respond to DMA controller messages and signals. In general, a host processor interacts with the DMA controller to control and synchronize the flow of data between devices and memories in the system in such a way as to avoid overrun and underrun conditions at the sources and destinations of data transfers.
The present invention provides a set of flexible addressing modes for supporting efficient data transfers to and from multiple memories, together with mechanisms for allowing data accesses to be directed to PEs according to virtual as opposed to physical IDs. This section describes an exemplary DMA controller and its system environment that provides one context in which the present invention may be effectively used. The discussion below addresses PE memory addressing, virtual-to-physical PE ID translation and its purpose, and a set of PE memory addressing modes or "PE addressing modes" which support numerous parallel algorithms and processes with highly efficient data transfer.
Fig. 2 shows an exemplary system 200 which illustrates the context in which a ManArray DMA controller 201, in accordance with the present invention, resides. The DMA controller 201 accesses processor local memories 210, 211, 212, 213, 214 and 215 via the DMA bus 202, 202h 2022, 2023, 2024, 2025 and the memory interface units 205, 206, 207, 208 and 209 to which it is connected. A ManArray DSP 203 also connects to its local memories 210-215 via memory interface units 205-209. Further details of a presently preferred DSP 203 are found in the above incorporated by reference applications. Although a ManArray DSP 203 is used in the preferred embodiment, the invention described herein may be used with any other processor which is coupled to an array or set of local memories such that the DMA controller has similar access capabilities.
In this representative system 200, the DMA controller 201 also connects to two system busses, a system control bus (SCB) 235, and the System Data Bus (SDB) 240. The DMA controller 201 is designed to transfer data between devices on the SDB 240, such as the system memory 250 and the DSP 203 local memories 210-215. The SCB 235 is used by an SCB master, such as the DSP 203 or a host control processor (HCP) 245, to access various system and DMA control registers. The DMA control registers are provided for initiating transfer or semaphore control operations and for reading transfer controller status. The SCB 235 is also used by the DMA controller 201 to send synchronization messages to other SCB bus slaves such as DSP control registers 225 and host I/O block 255. Some of the DSP control registers 225 act as "mailboxes" for receiving messages which may be sent by a DMA transfer controller or by another SCB master such as the HCP.
Fig. 3 shows a system 300 which illustrates the operation of a DMA controller 301 which may suitably be a multiprocessor specialized to carry out data transfers utilizing one or more transfer controllers 302 and 303. Each transfer controller can operate as an independent processor or work together with other transfer controllers to carry out data transfers. DMA busses 305, 310 provide, in the presently preferred embodiment, independent data paths to processor local memories 320, 321, 322, 323, 324, 325, one bus "lane" consisting of address, control, and data signals for each of the transfer controllers 302 and 303. In addition, each transfer controller is connected to an SDB 350 and to an SCB 330. Each transfer controller operates as a bus master and a bus slave on both the SCB and SDB. As a bus slave on the SCB, a transfer controller may be accessed by other SCB bus masters in order to read its internal state or issue control commands. As a bus master on the SCB, a transfer controller can send synchronization messages to other SCB bus slaves. As a bus master on the SDB, a transfer controller performs data reads and writes from or to system memory or I/O devices which are bus slaves on the SDB. As a bus slave on the SDB, a transfer controller can cooperate with another SDB bus master in a "slave mode" allowing the bus master to read or write data directly from or to its data FIFOs (as discussed further below). It should be noted that the DMA busses 305 and 310, the SDB 350, and the SCB 330 may be implemented in different ways. For example, they may be implemented with varying bus widths, protocols, or the like consistent with the teachings of the present invention.
Fig. 4 shows a system 400 having a single transfer controller 401 comprising a set of execution units including an instruction control unit (ICU) 440, a system transfer unit (STU) 402, a core transfer unit (CTU) 408 and an event control unit (ECU) 460. The terms "core" and "local" in the text are used as adjectives to distinguish resources or components which are primarily used by and directly connected to the "core" processor versus those which are primarily used by or connected to a host control processor. The term "system" is used as an adjective when referring to resources used primarily by the host control processor. An inbound data queue (IDQ) 405 is a data FIFO buffer which is written with data from an SDB 470 under control of the STU 402. Data is read from the IDQ 405 under control of the CTU 408 to be sent to local memories 430, or sent to the ICU 440 in the case of instruction fetches. An outbound data queue (ODQ) 406 is a data FIFO which is written with data from DMA busses 425 under control of the CTU 408, to be sent to an SDB 470 device or memory under the control of the STU 402. The CTU 408 may also read DMA instructions from any one of the local memories attached to the DMA busses 425. These DMA instructions are then forwarded to the ICU 440 for initial decoding. The ECU 460 receives signal inputs from external devices 465, commands from the SCB 450, and instruction data from the ICU 440. It generates output signals 435, 436 and 437 which may be used to generate interrupts on host control processors within the system, and can act as a bus master on the SCB 450 to send synchronization messages to SCB bus slaves. Each transfer controller within a ManArray DMA controller is designed to fetch its own stream of DMA instructions. DMA instructions are of five basic types: transfer; branch; load; synchronization; and state control. The branch, load, synchronization and state control types of instructions are collectively referred to as "control instructions", and distinguished from the transfer instructions which actually perform data transfers. DMA instructions in the exemplary system are typically of multi-word length and require a variable number of cycles to execute in order to minimize DMA instruction storage requirements. It will be apparent to those skilled in the art that other DMA instruction encodings, such as fixed-length encodings, might be chosen without departing from the teachings of the present invention. Although the preferred embodiment supports multiple DMA instruction types as described in further detail in U.S. Patent Application Serial No. 09/471,217 entitled "Methods and Apparatus for Providing Data Transfer Control" filed December 23, 1999 and incorporated by reference in its entirety herein, the present invention focuses on instructions and mechanisms which provide for flexible and efficient data transfers to and from multiple memories, including bit- reversed addressing across multiple PEs, multicast, and packing-gather operations. Referring further to system 400 of Fig. 4, transfer-type instructions are dispatched by an ICU 440 for further decoding and execution by an STU 402 and a CTU 408. A "transfer- system-inbound" or TSI instruction moves data from SDB 470 to IDQ 405 and is executed by the STU 402. A "transfer-core-inbound" (TCI) instruction moves data from the IDQ 405 to the DMA Bus 425 and is executed by the CTU. A "transfer-core-outbound" (TCO) instruction moves data from the DMA Bus 425 to the ODQ 406 and is executed by the CTU. A "transfer-system-outbound" (TSO) instruction moves data from the ODQ 406 to an SDB 470 and is executed by the STU. Two transfer instructions are required to move data between an SDB system memory and one or more SP or PE local memories on the DMA bus, and both instructions are executed concurrently: a TSI, TCI pair or a TSO, TCO pair. The address parameter of the STU transfer instructions, TSI and TSO, refers to addresses on the SDB while the address parameter of the CTU transfer instructions, TCI and TCO, refers to addresses on the DMA bus which target local memories.
Fig. 5 shows an exemplary instruction format 500 for transfer instructions. A base opcode field 501 indicates that the instruction is of transfer type. A C/S bit 510 indicates the transfer unit (CTU or STU) and I/O bit 520 indicates whether the transfer direction is inbound or outbound. Execute ("X") bit 550 when set to 1, indicates that the transfer should start immediately after decoding the transfer instruction. The execute bit provides one means of specifying a "start transfer" event. When the X bit is 0, the transfer parameters are decoded and loaded into the specified transfer unit, either the STU or the CTU, but the transfer is not initiated. Instruction fetch/decode continues normally until a "start transfer" event occurs. Data type field 530 indicates the size of each element transferred and address mode 540 refers to the data access pattern which must be generated by the transfer unit. A transfer count 560 indicates the number of data elements of size "data type" which are to be transferred to or from the target memory/device before an EOT occurs for that unit. An address parameter 570 specifies the starting address for the transfer. Other parameters 580 may follow the address word of the instruction, depending on the addressing mode used.
While there are six memories 210, 211, 212, 213, 214, and 215, shown in Fig. 2, the PE address modes access only the set of PE memories 210, 211, 212, and 213 in this exemplary ManArray DSP configuration. The address of a data element within PE local memory space is specified with three variables, a PE ID, a base value, and an index value. The base and the index values are summed to form a virtual offset into a PE memory relative to an address 0, the first address of that PE's memory. This virtual offset is then translated, via an address permutation and selection mechanism into a physical offset. The permutation and selection mechanism, together with PE ID translation, as described below, is designed to support data reorderings required for computations, such as FFT computations. The address of a PE data element may be thought of as a function of both the PE ID and the permuted sum of a base value and an index value:
PE data address = (PE ID, BitReversalSelect(Base + Index)), where the function "BitReversalSelect" is the permutation and selection function required to support FFT data reorderings within each local PE memory. The ManArray architecture incorporates a unique interconnection network between processing elements (PEs) which uses PE virtual IDs (VIDs) to support useful single-cycle communication paths, for example, torus and hypercube paths. In some array organizations, the PE's physical and virtual IDs are equal. The VIDs are used in the ManArray architecture to specify the pattern for data distribution and collection. When data is distributed according to the pattern established by VID assignment, then efficient inter-PE communication required by the programmer becomes available. To this end, Fig. 6 shows an exemplary table 600 illustrating one particular VID-to-PID translation. Fig. 7 shows an exemplary logical implementation of a system 700 for VID-to-PID translation. Fig. 8 shows an exemplary PExLAT instruction 800. Fig. 9 shows an exemplary VID-to-PID translation table register 900. Such aspects of a presently preferred embodiment are described in further detail in U.S. Patent Application Serial No. 09/472,372 entitled "Methods and Apparatus for Providing Direct Memory Access Control" and filed December 23, 1999.
For example, if a programmer needs to establish a hypercube connectivity for a 16 PE ManArray processor, the data will be distributed to the PEs according to a VID assignment in such a manner that the physical switch connections allow data to be transferred between PEs as though the switch topology were a hypercube even if the switch connections between physical PEs do not support the full hyper-cube interconnectivity. The present invention describes two approaches whereby the DMA controller can access PE memories according to their VIDs, effectively mapping PE virtual IDs to PE physical IDs (PIDs). The first uses VID-to-PID translation within the CTU of a transfer controller. This translation can be performed either through table-lookup, or through logic permutations on the VID. The second approach associates a VID with a PE by providing a programmable register within the PE or a PE local memory interface unit (LMIU) such as LMIUs 205, 206, 207 and 208 of Fig. 2, which is used by the LMIU logic to "capture" a data access when its VID matches a VID provided on the DMA Bus for each DMA memory access.
Bit-reverse PE addressing allows efficient scatter and gather of FFT data and coefficients. The DMA controller provides an efficient means for post-processing FFT calculations through its bit-reverse addressing capability. "Bit reversal" is a transposition of bits where the most significant bit (of a given "field" width) becomes least significant, and so on. For example, 0001011 will become 0011010 when the field width is 5 bits. "Digit reversal" is a transposition of groups of bits (a group of bits defines a digit) where the most significant digit becomes least significant and vice versa. For example, 000101 1 will become 0111000 for field width 6 and digit width 2.
In general, an FFT and similar algorithms are faster when they produce out of order output. However, one can implement FFTs that preserve the ordering with some additional cost. The reordering depends upon the radix used: Radix I Reversal digit width 2 1
4 2 8 3
2k k
Algorithms with radix larger than 8 seem to be impractical for a number of reasons, so typically only digit widths 1-3 are encountered. Also, any algorithm is capable in processing its own output, meaning that if algorithm A takes in-order input and produces some of the above reversals, then for the inverse transform, algorithm A with a reversed input of the same kind, will produce an in-order output. Complications arise when one is faced with providing or dealing with a reversed ordering that is not the result of the algorithm at hand. For example, if an algorithm is required to: 1) take a distributed reversed output from the PEs and generate an in-order vector in external memory, or 2) take a distributed reversed output from the PEs and generate a bit-reversed vector, or 3) provide the inverse of either of the above two.
The offset or vector index can be considered to consist of two fields: a) the distributed address (usually this is the PE id) consisting of the most significant bits, and b) the local address (rest of the bits). To achieve 1) above, bit-reverse PE IDs and then digit reverse local address according to radix.
To achieve 2) above, only bit-reverse within the digits.
To specify a bit-reversal DMA operation, a "bit reversal code" is stored in a DMA register, BITREV, whose default value is 0 specifying no reversal. This register is read/writeable from the MCB. There is one BITREV register per transfer controller and this register is reset to 0 on DMA RESET. A presently preferred BITREV register encoding format 1000, is shown in Fig. 10. The exemplary system supports bit-reverse addressing for radix 2, 4, and 8 FFTs, which corresponds to digit widths of 1,2, and 3 bits respectively. The system also supports reversed orderings (1) and (2) above and FFT sizes 256, 512, 1024, 2048, 4096 and 8192 that use 6, 7, 8, 9, 10, and 11 bits per PE address, respectively These bit requirements are defined in the exemplary system to support 32-bit word addressing on the DMA busses so that the least significant 2 bits are assumed to be zero.
Fig. 11 shows an exemplary encoding table 1100 showing the encodings and purpose for each valid BitRev code for the encoding format 1000 of Fig. 10. More specifically, Fig. 11 shows how to program the BITREV register to achieve the bit-reversed address operation for different sizes of FFTs and the number of PEs N. For example, a radix 2 FFT of size 4096 to be processed on an N=4 (2x2) array would use (4096/4=1024) radix 2 line entry requiring BITREV to be set to 0x05 as shown on line 1105. If the same radix 2 and 4096 point FFT is to be processed on an N=8 (2x4) array the (4096/8=512) radix 2 line entry would be used requiring BITREV to be set to 0x04 as seen on line 1106. PEXLAT must also be loaded to achieve the specified bit-reversed address operation. PEXLAT is a register that may be loaded either by MCB write access or by a DMA instruction. It specifies the order in which PEs are accessed for PE addressing modes. Note for Cooley-Tukey FFTs PEXLAT for inbound transfers is {0,1,2,3} and for outbound transfers the ordering is {0,2,1,3}. Details of an exemplary PEXLAT instruction 800 are shown in Fig. 8. Note for this example that PE address bits [1:0] are always assumed to be zero since they are not routed to the PEs, since DMA transfers, in the present exemplary system, are only in 32-bit words. The PE address bits for each valid BitRev code encoding with their corresponding (triple) are shown in table 1200 of Fig. 12. Using a 5-bit "bit reversal" code, which is always fed into a PE address output multiplexer, there are 17 possible variations as shown in Fig. 12, with a code value of 00000 corresponding to "no reversal" of bits.
A PE operation code refers to a set of signals and cycle types on the DMA bus which specify operations to be performed at the local memory interface unit (LMIU), or within the PEs themselves. In particular, this signal group is used to specify operations such as:
Start-Of-Sequence, End-Of-Sequence,
A specific PE which is to send/receive data,
A group of 2 or more PEs which are to receive data,
A group of PEs which are to send data on specific data lines of the DMA Bus (each PE uses a set of wires depending on its PE ID using virtual or physical IDs, depending on implementation,
A group of PEs which are to read data from specific data lines of the DMA Bus (each PE uses a set of wires depending on its PE ID (virtual or physical, depending on implementation),
DMA virtual channel ID used to control the update of semaphores associated with local memory access by PEs or DMA lane, or
Other operations best carried out in a PE-relative fashion, or at the local memories.
One basic idea is that a small group of signals included in the opcode bus can be used to specify things like:
Load PE Opcode - a bit that says that corresponding data is a PE Opcode, Enable PE Opcode - a bit that enables the use of the PE Opcode in LMIU on current data,
Disable PE Opcode - a bit that disables use of the PE opcode in LMIU on current data,
Data Operation - a bit that specifies that the access is a data operation using current setting of PE Opcode (if not disabled), or
PE ID - indicates a specific PE address for normal data accesses (no PE Opcode used).
In addition, it is noted that using a small set of signals, the data bus can be used to specify a PE opcode. One of the bits in the set of signals indicates that the cycle type is "Load new PE Opcode". The PE opcode state is disabled when the last cycle of a transfer is presented to the LMIU. PE Multicast Addressing
PE multicast addressing refers to parallel distribution of the same data element to more than one, and up to all, PEs simultaneously. In other words, for the all PE case it refers to a "broadcast" operation. Multicast addressing is specified as a set of addressing modes in the TCI instruction, as in the address mode field 540 of Fig. 5. In addition, another parameter in the transfer instruction specifies which PEs are to accept the data transfer. Three exemplary approaches to such specification are as follows:
A 16 bit field can support the specification of any combination of up to 16 PEs to receive the same data element. A 4 bit number can specify any set of 16 PEs to receive the same data element when used in combination with the PE VID-to-PID translation table. The 4 bit number N, where '0' implies the value N ='16', specifies that the first N elements in the translate table are to be selected to receive the data element. Since the translate table can specify PEs in any order, any group of PEs may be selected to receive the data. An encoded M-bit value can be used to specify selected, common combinations of
PEs to receive data simultaneously.
It is further noted that multicast addressing can be used along with block, stride and circular addressing to any group of PEs.
The PE relative packing-gather operations type indicates that each PE drives data onto a different group of data wires to return to the transfer controller ODQ. For example, a byte size gather over 4 PEs would specify that each PE supplies one byte of a 32-bit word to be returned to the ODQ for transfer to the system data bus. Other data types may be supported, with varying numbers of PEs participating in the gather operation.
In a PE relative gather-sum operations, a specified number of data elements are summed as they are read from local memories. If the summing parameter is N, then a single data element is transferred to the ODQ for every N elements read from local memories. The N elements are summed with the result being placed in the ODQ for transfer to the system data bus.
The PE Relative Unpack-Distribute Operations type indicates that each PE LMIU receives data from a different group of data wires to be written to its local memory. For example, a byte size unpack-distribute over 4 PEs would specify that each PE reads one byte of a 32-bit word to be subsequently written to the local memory. Optional zero-extend, sign-extend or one- extend may be supported. Other data types may be supported, such as halfwords, with varying numbers of PEs participating in the gather operation. The DMA Bus width influences the number of useful data types which might be supported. Additionally, data may be permuted before being sent to PEs for inbound transfers, or before being sent to system memories for outbound transfers. Besides reordering of data within a data element, other stream oriented operations may be performed including masking, data merging such as AND with a mask followed by OR with a constant, complementation using XOR with a specified mask.
While the present invention is disclosed in a presently preferred context, it will be recognized that the teachings of the present invention may be variously embodied consistent with the disclosure and claims. By way of example, the present invention is disclosed in connection with specific aspects of the ManArray architecture. It will be recognized that the present teachings may be adapted to other present and future architectures to which they may be beneficial.

Claims

We claim:
1. An array processing system comprising: a digital signal processor (DSP); a direct memory access (DMA) controller; a plurality of processing element (PE) local memories; a direct memory access bus utilized by the DMA controller to access the plurality of PE local memories; a system memory; a system control bus (SCB) connected to the DMA controller; and a system data bus (SDB) connecting the DMA controller and a plurality of devices, wherein the DMA controller transfers data between the plurality of devices on the SDB.
2. The array processing system of claim 1 wherein the system memory is on the SDB and the plurality of PE local memories are on a direct memory access bus as controlled by the DMA controller to move data between the system memory and the plurality of PE local memories.
3. The array processing system of claim 1 wherein the DSP functions as an SCB master and utilizes the SCB to program the DMA controller with read and write addresses and register values to initiate control operations and read status.
4. The array processing system of claim 1 further comprising a host processor connected to the SCB, and which both functions as an SCB master and utilizes the SCB to program the DMA controller with read and write addresses and register values to initiate control operations and read status.
5. The array processing system of claim 1 wherein the DMA controller further utilizes the SCB to send synchronization messages to other SCB bus slaves.
6. The array processing system of claim 5 wherein the other SCB bus slaves comprise DSP control registers or a host input/output block.
7. The array processing system of claim 5 further comprising a host processor connected to the SCB, and wherein the DSP or the host processor can poll registers in bus slaves on the SCB to receive status data from the DMA controller.
8. The array processing system of claim 1 wherein the DMA controller operates to perform write operations to slave addresses which are programmed to cause interrupt side effects to the DSP allowing DMA controller messages to be handled by interrupt service routines.
9. A direct memory access (DMA controller for supporting efficient data transfers concurrent with host processor computation comprising: a first transfer controller which can operate as an independent processor or can work together with another transfer controller to carry out data transfers; a system control bus (SCB) connected to the transfer controller; a system data bus (SDB) connected to the transfer controller; the first transfer controller operating as both a bus master and a bus slave on both the
SCB and the SDB.
10. The DMA controller of claim 9 further comprising a second transfer controller which can operate as an independent processor or can work together with another transfer controller to carry out data transfers.
11. The DMA controller of claim 9 further comprising a first DMA bus independently connecting the first controller to a plurality of local memories.
12. The DMA controller of claim 10 further comprising a second DMA bus independently connecting the second transfer controller a plurality of local memories.
13. The DMA controller of claim 11 wherein the plurality of local memories comprise separate data random access memories (RAM) for at least four processing elements (PEs), data RAM for a sequence processing element and instruction RAM for the sequence processing element.
14. A transfer controller comprising: a set of execution units including an instruction control unit (ICU), a system transfer unit (STU), a core transfer unit (CTU), and an event control unit (ECU); an inbound data queue (IDQ) comprising a data first in first out buffer; an outbound data queue (ODQ) comprising a data first in first out buffer; a system data bus (SDB); a system control bus (SCB); and a direct memory access (DMA) bus.
15. The transfer controller of claim 14 wherein the STU is connected to the system data bus (SDB) and controls the writing of data from the SDB to the IDQ, data is read from the IDQ under control of the CTU to be sent through the DMA bus to core memories or to the ICU in the core of instruction fetches.
16. The transfer controller of claim 14 wherein the CTU is connected to the DMA bus and data from the DMA bus is written to the ODQ under control of the CTU to be sent through the SDB to an SDB device or memory under control of the STU.
17. The transfer controller of claim 14 wherein the CTU is connected to the DMA bus and reads DMA instructions from a memory attached to the DMA bus and forwards these read instructions to the ICU for initial decoding.
18. The transfer controller of claim 14 wherein the ECU is connected to receive signals from external devices.
19. The transfer controller of claim 14 wherein the ECU is connected to the SCB and receives commands from the SCB.
20. The transfer controller of claim 14 wherein the ECU is connected to the ICU and receives instruction data from the ICU.
21. The transfer controller of claim 14 wherein the ECU generator output signals which may be used to generate interrupts on a host control processor in a system including said transfer controller.
22. The transfer controller of claim 14 wherein said controller fetches its own stream of DMA instructions.
23. The transfer controller of claim 22 wherein said DMA instructions include transfer, branch, load, synchronization and status control instructions.
24. The transfer controller of claim 14 wherein transfer-type instructions are fetched by the ICU and dispatched for further decoding and execution by the STU and CTU.
25. The transfer controller of claim 24 wherein a transfer-system-inbound instruction moves data from the SDB to the IDQ and is executed by the STU.
26. The transfer controller of claim 24 wherein a transfer-core-outbound instruction moves data from the DMA bus to the ODQ and is executed by the CTU.
27. The transfer controller of claim 24 wherein a transfer-system-outbound instruction moves data from the ODQ to the SDB and is executed by the STU.
28. The transfer controller of claim 24 wherein two transfer instructions are required to move data between an SDD system memory and one or more SP or PE local memories on the DMA bus, and both transfer instructions are executed concurrently.
29. The transfer controller of claim 24 wherein the transfer-type instructions include an address parameter which is decoded by the STU and which refers to an address on the SDB.
30. The transfer controller of claim 14 wherein control-type instructions include an address parameter which is decoded by the core transfer unit and which refers to addresses on the DMA bus to processing element and sequence processor local memories.
31. An instruction format for a transfer instruction comprising: a base opcode field indicating that an instruction is of transfer type; a C/S filled which indicates a type of transfer unit for the instruction, a control transfer unit (CTU) or a system transfer unit (STU), respectively; an I/O field which indicates that the instruction is inbound or outbound, respectively; an execute (X) field when set indicates a start transfer event should start immediately after loading the instruction, and when not set indicates that the instruction is loaded but transfer is not initiated; a data type field indicates the size of each element transferred; an address mode field refers to a data access pattern which must be generated by a transfer unit, either the CTU or STU; a transfer count field indicates the number of data elements of the size specified by the data type field are to be transferred before an end of transfer occurs; and an address parameter field which specifies a starting address for the transfer.
32. The instruction format of claim 31 further comprising: an other parameter field which follows the address parameter field depending on the addressing mode used.
33. The instruction format of claim 31 wherein an address of a data element within a PE local memory space is specified with three variables, a PE ID, a base value and an index value.
34. The instruction format of claim 31 wherein to support data reorderings the address of a PE data element is specified by a pair: PE data address = (PE ID, BitReversalSelect (Base & Index)), where the function BitReversalSelect is a permutation and selection function to support fast Fourier transform (FFT) data reorderings within each local processing element (PE) memory.
35. A method for multicast addressing to enable parallel distribution of a data element to more than one and up to all processing elements (PEs) in a system simultaneously comprising the steps of: specifying an addressing mode in an address mode field of a transfer control instruction (TCI); and utilizing another parameter in the TCI to specify which PEs are to accept data transfer.
36. The method of claim 35 further comprising the step of: utilizing a 16 bit field to specify any combination of up to 16 PEs to receive the data element.
37. The method of claim 35 further comprising the step of: utilizing a 4 bit number to specify any set of 16 PEs to receive the data element, the 4 bit number being used in combination with a PE VID-to-PID translation table to specify the PEs to receive the data element.
38. The method of claim 35 further comprising the step of: utilizing an encoded M-bit value to specify the PEs to receive the data element simultaneously.
39. A method for permuting data before the data is sent to processing elements (PE) for inbound transfers or before being sent to system memories for outbound transfers comprising the steps of: reordering data within a data element; and performing other stream oriented operations including masking, data merging or complementing.
40. The method of claim 39 wherein said step of data merging further comprises: performing a logical AND operation with a mask followed by performing a logical
OR operation with a constant.
41. The method of claim 39 wherein said step of complementing further comprises using a logical XOR operation with a specified mask.
42. A method for performing processing element (PE) packing-gather operations comprising the following steps: setting a packing-gather operations type indicator to indicate that each PE drives data onto a different group of data wires to return to a transfer controller ODQ; and determining the packing-gather operations type indicator to control PE deriving of data.
43. The method of claim 42 wherein a byte size gather over 4 PEs specifies that each PE supplies one byte of a 32-bit word to be returned to the ODQ for return to the system data bus.
44. A method for performing processing element (PE) relative gather-sum operations comprising: specifying a summary transfer parameter, N, specifying a number of data elements which are to be summed as they are read from local memories; summing the N elements to form a single data element; repeating the previous steps; and transferring a single data element to an ODQ as a sum result for every N elements read from local memories.
45. The method of claim 44 further comprising the step of transferring the sum result from the ODQ to a system data bus.
46. A method for performing processing element (PE) relative unpack-distribute operations comprising the steps of: setting an unpack-distribute operations type indicator to indicate that each PE LMIU receives data from a different group of data wires to be written to its local memory; and determining the unpack-distribute operations type indicator to control each PE LMIU's receipt of data.
PCT/US2001/005714 2000-02-24 2001-02-23 Methods and apparatus for providing bit-reversal and multicast functions utilizing dma controller WO2001063438A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US18466800P 2000-02-24 2000-02-24
US60/184,668 2000-02-24

Publications (1)

Publication Number Publication Date
WO2001063438A1 true WO2001063438A1 (en) 2001-08-30

Family

ID=22677854

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2001/005714 WO2001063438A1 (en) 2000-02-24 2001-02-23 Methods and apparatus for providing bit-reversal and multicast functions utilizing dma controller

Country Status (2)

Country Link
US (9) US6834295B2 (en)
WO (1) WO2001063438A1 (en)

Families Citing this family (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2002093389A1 (en) * 2001-05-17 2002-11-21 Decru, Inc. A stream-oriented interconnect for networked computer storage
KR100606163B1 (en) 2004-07-10 2006-08-01 삼성전자주식회사 Direct memory access device, system and method for transmitting and receiving data through direct memory access device
GB2433333B (en) * 2005-12-13 2011-07-13 Advanced Risc Mach Ltd Distributed direct memory access provision within a data processing system
GB2433611A (en) * 2005-12-21 2007-06-27 Advanced Risc Mach Ltd DMA controller with virtual channels
US8954943B2 (en) * 2006-01-26 2015-02-10 International Business Machines Corporation Analyze and reduce number of data reordering operations in SIMD code
US11834623B2 (en) 2006-12-21 2023-12-05 Delcotto Ip, Llc Wooden wicks including a booster for a candle and method of making
US8708694B2 (en) 2006-12-21 2014-04-29 Dream Wick Inc. Wooden wicks including a booster for a candle and method of making
KR100881191B1 (en) * 2007-03-27 2009-02-05 삼성전자주식회사 Apparatus for Multi Protocol Serial Interface and System On Chip thereof
FR2918190B1 (en) * 2007-06-26 2009-09-18 Thales Sa ADDRESSING DEVICE FOR PARALLEL PROCESSOR.
US20100146518A1 (en) * 2008-12-08 2010-06-10 International Business Machines Corporation All-To-All Comparisons on Architectures Having Limited Storage Space
US9274851B2 (en) 2009-11-25 2016-03-01 Brocade Communications Systems, Inc. Core-trunking across cores on physically separated processors allocated to a virtual machine based on configuration information including context information for virtual machines
US20110228772A1 (en) * 2010-03-19 2011-09-22 Brocade Communications Systems, Inc. Providing multicast services without interruption upon a switchover
US8959278B2 (en) 2011-05-12 2015-02-17 Freescale Semiconductor, Inc. System and method for scalable movement and replication of data
US8554963B1 (en) * 2012-03-23 2013-10-08 DSSD, Inc. Storage system with multicast DMA and unified address space
GB2500707B (en) * 2012-03-30 2014-09-17 Cognovo Ltd Multiprocessor system, apparatus and methods
US10581763B2 (en) 2012-09-21 2020-03-03 Avago Technologies International Sales Pte. Limited High availability application messaging layer
US9967106B2 (en) 2012-09-24 2018-05-08 Brocade Communications Systems LLC Role based multicast messaging infrastructure
US10095433B1 (en) 2012-10-24 2018-10-09 Western Digital Technologies, Inc. Out-of-order data transfer mechanisms for data storage systems
US9323654B2 (en) * 2013-07-17 2016-04-26 Infineon Technologies Ag Memory access using address bit permutation
GB2529425A (en) 2014-08-19 2016-02-24 Ibm Data processing apparatus and method
US9619349B2 (en) 2014-10-14 2017-04-11 Brocade Communications Systems, Inc. Biasing active-standby determination
US9816053B2 (en) 2015-03-26 2017-11-14 Melynda S DelCotto Candle having a wooden wick with figured grain

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4516199A (en) * 1979-10-11 1985-05-07 Nanodata Computer Corporation Data processing system
US5166674A (en) * 1990-02-02 1992-11-24 International Business Machines Corporation Multiprocessing packet switching connection system having provision for error correction and recovery
US5317715A (en) * 1987-12-15 1994-05-31 Advanced Micro Devices, Inc. Reduced instruction set computer system including apparatus and method for coupling a high performance RISC interface to a peripheral bus having different performance characteristics
US5701482A (en) * 1993-09-03 1997-12-23 Hughes Aircraft Company Modular array processor architecture having a plurality of interconnected load-balanced parallel processing nodes
US5812876A (en) * 1994-04-25 1998-09-22 Compaq Computer Corporation DMA controller which can be controlled by host and local processors
US6173393B1 (en) * 1998-03-31 2001-01-09 Intel Corporation System for writing select non-contiguous bytes of data with single instruction having operand identifying byte mask corresponding to respective blocks of packed data
US6173388B1 (en) * 1998-04-09 2001-01-09 Teranex Inc. Directly accessing local memories of array processors for improved real-time corner turning processing

Family Cites Families (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE69132300T2 (en) * 1990-03-12 2000-11-30 Hewlett Packard Co Direct memory access defined by the user using virtual addresses
US5664142A (en) * 1990-10-01 1997-09-02 International Business Machines Corporation Chained DMA devices for crossing common buses
US5864738A (en) * 1996-03-13 1999-01-26 Cray Research, Inc. Massively parallel processing system using two data paths: one connecting router circuit to the interconnect network and the other connecting router circuit to I/O controller
US6167502A (en) * 1997-10-10 2000-12-26 Billions Of Operations Per Second, Inc. Method and apparatus for manifold array processing
US6449664B1 (en) * 1998-11-16 2002-09-10 Viewahead Technology, Inc. Two dimensional direct memory access in image processing systems
US6260082B1 (en) * 1998-12-23 2001-07-10 Bops, Inc. Methods and apparatus for providing data transfer control
US6256683B1 (en) * 1998-12-23 2001-07-03 Bops, Inc. Methods and apparatus for providing direct memory access control
US6715066B1 (en) * 2000-04-07 2004-03-30 Sun Microsystems, Inc. System and method for arranging bits of a data word in accordance with a mask
US6713393B2 (en) * 2002-06-20 2004-03-30 Intelligent Sources Development Corp. Method of forming a nanometer-gate MOSFET device
US20060088046A1 (en) * 2004-10-26 2006-04-27 Wong Kar L Queue resource sharing for an input/output controller
US7424565B2 (en) * 2005-11-16 2008-09-09 Sun Microsystems, Inc. Method and apparatus for providing efficient output buffering and bus speed matching
US8437369B2 (en) * 2006-05-19 2013-05-07 Integrated Device Technology, Inc. Packets transfer device that intelligently accounts for variable egress channel widths when scheduling use of dispatch bus by egressing packet streams
US7657663B2 (en) * 2006-12-19 2010-02-02 International Business Machines Corporation Migrating stateless virtual functions from one virtual plane to another
US7707346B2 (en) * 2008-03-12 2010-04-27 Lsi Corporation PCI express multi-root IOV endpoint retry buffer controller

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4516199A (en) * 1979-10-11 1985-05-07 Nanodata Computer Corporation Data processing system
US5317715A (en) * 1987-12-15 1994-05-31 Advanced Micro Devices, Inc. Reduced instruction set computer system including apparatus and method for coupling a high performance RISC interface to a peripheral bus having different performance characteristics
US5166674A (en) * 1990-02-02 1992-11-24 International Business Machines Corporation Multiprocessing packet switching connection system having provision for error correction and recovery
US5701482A (en) * 1993-09-03 1997-12-23 Hughes Aircraft Company Modular array processor architecture having a plurality of interconnected load-balanced parallel processing nodes
US5812876A (en) * 1994-04-25 1998-09-22 Compaq Computer Corporation DMA controller which can be controlled by host and local processors
US6173393B1 (en) * 1998-03-31 2001-01-09 Intel Corporation System for writing select non-contiguous bytes of data with single instruction having operand identifying byte mask corresponding to respective blocks of packed data
US6173388B1 (en) * 1998-04-09 2001-01-09 Teranex Inc. Directly accessing local memories of array processors for improved real-time corner turning processing

Also Published As

Publication number Publication date
US20020002640A1 (en) 2002-01-03
US7765338B2 (en) 2010-07-27
US6834295B2 (en) 2004-12-21
US20050038936A1 (en) 2005-02-17
US20110302333A1 (en) 2011-12-08
US20120331185A1 (en) 2012-12-27
US20100257290A1 (en) 2010-10-07
US20080016262A1 (en) 2008-01-17
US7975080B2 (en) 2011-07-05
US20050289259A1 (en) 2005-12-29
US6986020B2 (en) 2006-01-10
US8082372B2 (en) 2011-12-20
US20140075081A1 (en) 2014-03-13
US8244931B2 (en) 2012-08-14
US20110225392A1 (en) 2011-09-15
US8601176B2 (en) 2013-12-03
US9442872B2 (en) 2016-09-13

Similar Documents

Publication Publication Date Title
US9442872B2 (en) Methods and apparatus for providing bit-reversal and multicast functions utilizing DMA controller
US20210149820A1 (en) Two address translations from a single table look-aside buffer read
US11429387B2 (en) Streaming engine with stream metadata saving for context switching
USRE41904E1 (en) Methods and apparatus for providing direct memory access control
US10606598B2 (en) Dual data streams sharing dual level two cache access ports to maximize bandwidth utilization
US11809362B2 (en) Superimposing butterfly network controls for pattern combinations
US20240063827A1 (en) Butterfly network on load data return
US10339057B2 (en) Streaming engine with flexible streaming engine template supporting differing number of nested loops with corresponding loop counts and loop offsets
US20230004391A1 (en) Streaming engine with stream metadata saving for context switching

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): CA IL JP KR

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
122 Ep: pct application non-entry in european phase
NENP Non-entry into the national phase

Ref country code: JP