US20070097239A1 - CMOS image sensor apparatus with on-chip real-time pipelined JPEG compression module - Google Patents

CMOS image sensor apparatus with on-chip real-time pipelined JPEG compression module Download PDF

Info

Publication number
US20070097239A1
US20070097239A1 US11/524,534 US52453406A US2007097239A1 US 20070097239 A1 US20070097239 A1 US 20070097239A1 US 52453406 A US52453406 A US 52453406A US 2007097239 A1 US2007097239 A1 US 2007097239A1
Authority
US
United States
Prior art keywords
image
cmos
module
data
fifo
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US11/524,534
Inventor
Michael Kaplinsky
Igor Subbotin
Xiuhong Lin
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Micron Technology Inc
Original Assignee
Micron Technology Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Micron Technology Inc filed Critical Micron Technology Inc
Priority to US11/524,534 priority Critical patent/US20070097239A1/en
Publication of US20070097239A1 publication Critical patent/US20070097239A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00General purpose image data processing
    • G06T1/20Processor architectures; Processor configuration, e.g. pipelining
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00Circuitry of solid-state image sensors [SSIS]; Control thereof

Definitions

  • the invention relates generally to improved semiconductor imaging devices and, more specifically, to a CMOS imager provided with an on-chip data compression module.
  • CMOS imagers A number of different types exist, including charge coupled devices (CCDs), CMOS arrays, photodiode arrays, charge injection devices and hybrid focal plane arrays. Recently, however, CMOS imagers have gained popularity in use in a wide variety of electronic devices, because CMOS imagers offer a number of advantages over other types of imagers. CMOS imagers, for example, are compatible with integrated on-chip electronics (control logic and timing, image processing, and signal conditioning such as A/D conversion). CMOS imagers allow random access to the image data. CMOS imagers have lower fabrication costs as compared with the conventional CCD imagers, since standard CMOS processing techniques can be used.
  • CMOS imagers have low power consumption because only one row of pixels at a time needs to be active during the readout and there is no charge transfer (and associated switching) from pixel to pixel during image acquisition.
  • On-chip integration of electronics is particularly advantageous because of the potential to perform many signal conditioning functions in the digital domain (versus analog signal processing) as well as to achieve a reduction in system size and cost.
  • CMOS imagers as discussed above are generally known as discussed, for example, in Nixon et al., “256 ⁇ 256 CMOS Active Pixel Sensor Camera-on-a-Chip,” IEEE Journal of Solid-State Circuits, Vol. 31(12) pp. 2046-2050, 1996; Mendis et al, “CMOS Active Pixel Image Sensors,” IEEE Transactions on Electron Devices, Vol. 41(3) pp. 452-453, 1994 as well as U.S. Pat. No. 5,708,263, U.S. Pat. No. 5,471,515, and U.S. Pat. No. 6,204,524, which are herein incorporated by reference.
  • CMOS image sensor technology Recent advances in CMOS image sensor technology include the integration of the imager and sophisticated image processing modules on a single die, as mentioned above. These systems-on-a-chip (SOCs) usually have cost, power consumption and form-factor advantages over the multi-chip solutions with the same functionality. Furthermore, image processing modules integrated with CMOS imagers can be fine tuned to the specific properties of the given imager and to the needs of the targeted applications.
  • One segment of the market where the low power of CMOS imagers is most advantageous is the mobile devices market. Many cell-phone designs are incorporating image sensors in their architectures. However, bandwidth limitations imposed by wireless transmission, together with the desire to employ image sensors with large pixel counts, necessitates the use of image compression in the system. Many of the existing designs draw on the ability of the on-board CPU to perform image compression, but at speeds far below video rates. Other available solutions rely on an additional image processing chip to perform color processing and compression.
  • the present invention provides a CMOS imager in which a CMOS image sensor, a color image processing module and an image compression module are all provided on a single die. Both the color image processing module and the image compression module incorporate pipelined architectures to process the image data at a video rate in a massively parallel fashion.
  • FIG. 1 is a block diagram of a typical CMOS pixel sensor chip.
  • FIG. 2 is a block diagram of a preferred embodiment of the present invention.
  • FIG. 3 is a block diagram of the baseline JPEG compression module of the present invention.
  • FIG. 4 is a block diagram of the SRAM Addressing Scheme of the present invention.
  • FIG. 5 is a block diagram of the FIFO & Register Control module of the present invention.
  • FIG. 6 is an illustration of a computer system having a CMOS imager according to the present invention.
  • FIG. 1 illustrates a block diagram for a typical CMOS imager 10 having a pixel array 100 with each pixel cell being constructed in the manner disclosed, for example, in U.S. Pat. No. 6,204,524, the disclosure of which is herein incorporated by reference.
  • Pixel array 100 comprises a plurality of pixels arranged in a predetermined number of columns and rows. The pixels of each row in array 100 are all turned on at the same time by a row select line, and the pixels of each column are selectively output by a column select line. A plurality of rows and column lines are provided for the entire array 100 .
  • the row lines are selectively activated by the row driver 110 in response to row address decoder 120 and the column select lines are selectively activated by the column driver 160 in response to column address decoder 170 .
  • a row and column address is provided for each pixel.
  • the CMOS imager is operated by the control circuit 150 which controls address decoders 120 , 170 for selecting the appropriate row and column lines for pixel readout, and row and column driver circuitry 110 , 160 which apply driving voltage to the drive transistors of the selected row and column lines.
  • CMOS imager In the operation of the FIG. 1 CMOS imager, one of the rows of pixels is selected for readout by using row driver 110 as described above. All pixels on a selected row are processed simultaneously and sampled onto a capacitor at the bottom of their respective columns.
  • the column-parallel sampling process typically takes 1-10 m ⁇ sec, and preferably occurs during the so-called horizontal blanking interval of a video image.
  • Each column is then successively selected for read-out of the voltage stored in the capacitor of that column using column driver 160 .
  • the output data from CMOS imager 10 is then processed in accordance with the present invention as described below.
  • color image processor 202 converts digitized imager data (output from CMOS imager 10 of FIG. 1 ) from its native format (such as RGB) into a format more suitable for image compression, such as YCrCb (according to ISO CCIR-601).
  • CMOS imager 10 of FIG. 1 its native format
  • YCrCb a format more suitable for image compression
  • the data stream is passed to highly pipelined compression module 204 .
  • compression module 204 implements JPEG compression at the video rate.
  • line buffers 206 , 208 at the front end of the compression designed to allow two-dimensional processing.
  • JPEG compression module 204 implements baseline JPEG compression, as described below in more detail with reference to FIG. 3 .
  • FIFO & Register & CPU Control module 205 implements all of the interface control and communicates with each of the modules and the outside CPU data bus.
  • FIG. 3 provides a more detailed block of baseline JPEG compression module 204 .
  • the circuitry is arranged in such a way as to pass the data from one stage to the next, allowing the entire compression operation to be performed at the video pixel rate.
  • JPEG compression module 204 divides up the image into 8 by 8 pixel blocks in module 300 , and then calculates the fast discrete cosine transform (FDCT) of each block in module 301 .
  • Quantizer module 302 rounds off the FDCT coefficients according to a specified quantization matrix. This step produces the “lossy” nature of JPEG, but allows for large compression ratios.
  • FDCT fast discrete cosine transform
  • Zigzag module 303 then performs variable length coding on these coefficients, followed by Huffman coding in module 305 , which then outputs the compressed binary bitstream.
  • JPEG recovers the quantized DCT coefficients from the compressed data stream, takes the inverse transforms and displays the image.
  • the present invention utilizes a unique SRAM addressing scheme to reduce the memory area and improve the utilization rate of memory to maximum.
  • the output from Color Image Processing module 202 is continually written to the corresponding SRAM 206 , 208 line by line.
  • JPEG compression module 204 works on 8*8 blocks of data, 8 lines of data have to be ready before compression.
  • the challenge is how to read and write to the SRAM at the same time.
  • the first scheme uses a “ping-pong” algorithm, which uses two SRAMs, one to store the data from Color Image Processing module, and the other for JPEG compression. This scheme switches SRAM banks after every interval of 8 lines of data.
  • the second scheme uses only one SRAM.
  • the JPEG compression module 204 begins to read after 8 lines of data have been written to the SRAM 206 , 208 . Then, the output from Color Image Processing module 202 is written to the location in the SRAM from where the data has just been read. Compared with the ping-pong scheme, the latter scheme uses only one-half of the amount of SRAM. In the present invention, the second scheme is utilized, thereby greatly reducing the chip area.
  • the SRAM uses the minimum size to hold 8 lines of image data for JPEG compression.
  • Module 401 automatically detects the video size and self-adjusts the addressing scheme.
  • Module 402 delays reading for eight lines so that full 8 lines of data are ready for JPEG processing.
  • Module 404 extracts two dimensional data from the linear SRAM, separates Y, Cb and Cr to reconstruct 8*8 blocks, and outputs the data in block order of Y Y Cb Cr, while new video data is simultaneously written to the SRAM. In each block, the data is output line by line.
  • Address_next Clip (Address_current + Delta_current)
  • Delta_next Delta_current * ( LineSize / 8 ) ⁇ ( LineSize ⁇ 1 ) * [ Delta_current * ( LineSize / 8) / ( LineSize ⁇ 1 ) ]
  • the output data rate of the encoder is variable (intermittent). This might present a problem for the overall system architecture, as in the absence of the frame buffer there would be a need to constantly monitor the availability of the compressed data for transfer to the system memory or direct transmission to the remote site. This type of system behavior would unnecessarily tie-up system resources (such as CPU cycles and data bus bandwidth).
  • the present invention incorporates a relatively small memory buffer, FIFO 210 , at the output of the encoder. This buffer, being much smaller that a full frame buffer, preserves the cost advantages of the system and allows for periods of fixed-rate data output interspersed with periods of inactivity.
  • output buffer 210 is a dual-ported memory together with pointer control block, allowing the buffer to function as a FIFO (first-in-first-out).
  • the pointer control block allows for storage of the output of the encoder as soon as it becomes available, while allowing independent retrieval of the data based on the external requests, as long as FIFO is not empty.
  • the FIFO control block generates Half-Full and/or Almost Empty/Over X bytes flags describing the state of the content of the FIFO. These signals can then be used as an interrupt for external controller, prompting data retrieval cycle and relieving the external controller from the need to constantly monitor data availability in the FIFO.
  • FIG. 5 shows a more detailed block diagram of the FIFO & Register & CPU Control module, which implements the communication with the other modules.
  • CPU Interface Control Module 502 communicates with the CPU data bus through control signals and bi-directional data bus for various operations, such as register read/write, FIFO data read, FIFO current usage, INT status, etc.
  • Decoder module 503 handles control signal decoding.
  • Module 506 generates an INT signal according to the current FIFO status. Reading or writing to a specified register via the Register Bus ( FIG. 2 ) is coordinated through Register Interface Control module 500 .
  • FIFO Interface Control module 504 controls the reading process from FIFO. All interfaces are controlled and synchronized by Overall Interface Control module 501 .
  • the INT signal is supplied to access the internal FIFO over the system data/memory bus.
  • the CPU To read video data correctly from the internal FIFO, the CPU must know the current state of FIFO. The following FIFO conditions will generate the INT signal transition and will also be reflected as the corresponding INT status register bits.
  • the CPU inquiries the INT status register to get the current FIFO states. Based on the FIFO status, the CPU initiates or stops the video data read process or processes the data.
  • Bit 4 X bytes in FIFO where X is programmable
  • the FIFO outputs are connected directly to the system data/memory bus, allowing for the access to the imager system to be performed over standard system communication channel in a way similar to the memory access.
  • the access to the registers/control functions of the imager system itself is also performed through the same bus interface.
  • the control signals are provided to allow distinguishing between various traffic over the output pins: image access, register write and register read access.
  • the imager system of the present invention may also need to provide uncompressed video stream either in full frame format or in decimated format (such as VGA image decimated to CIF resolution). Accordingly, in another embodiment of the present invention, the encoder output (compressed data stream) and the uncompressed video can be multiplexed to the input port of the output memory buffer (FIFO).
  • FIFO output memory buffer
  • a typical processor based system that includes a CMOS imager device according to the present invention is illustrated generally at 600 in FIG. 6 .
  • a processor based system is exemplary of a system having digital circuits which could include CMOS imager devices. Without being limiting, such a system could include a computer system, camera system, scanner, machine vision system, vehicle navigation system, video telephone, surveillance system, auto focus system, star tracker system, motion detection system, image stabilization system and data compression system for high-definition television, all of which can utilize the present invention.
  • a processor system such as a computer system, for example generally comprises a central processing unit (CPU) 644 that communicates with an input/output (I/O) device 646 over a bus 652 .
  • the CMOS imager 610 also communicates with the system over bus 652 .
  • the computer system 600 also includes random access memory (RAM) 648 , and, in the case of a computer system may include peripheral devices such as a floppy disk drive 654 and a compact disk (CD) ROM drive 656 which also communicate with CPU 644 over the bus 652 .
  • RAM random access memory
  • CMOS imager 610 is combined with a pipelined JPEG compression module in a single integrated circuit.

Abstract

A CMOS imager in which a CMOS image sensor, a color image processing module and an image compression module are all provided on a single die. Both the color image processing module and the image compression module incorporate pipelined architectures to process the image data at a video rate in a massively parallel fashion.

Description

    FIELD OF THE INVENTION
  • The invention relates generally to improved semiconductor imaging devices and, more specifically, to a CMOS imager provided with an on-chip data compression module.
  • BACKGROUND OF THE INVENTION
  • A number of different types of semiconductor-based imagers exist, including charge coupled devices (CCDs), CMOS arrays, photodiode arrays, charge injection devices and hybrid focal plane arrays. Recently, however, CMOS imagers have gained popularity in use in a wide variety of electronic devices, because CMOS imagers offer a number of advantages over other types of imagers. CMOS imagers, for example, are compatible with integrated on-chip electronics (control logic and timing, image processing, and signal conditioning such as A/D conversion). CMOS imagers allow random access to the image data. CMOS imagers have lower fabrication costs as compared with the conventional CCD imagers, since standard CMOS processing techniques can be used. Additionally, CMOS imagers have low power consumption because only one row of pixels at a time needs to be active during the readout and there is no charge transfer (and associated switching) from pixel to pixel during image acquisition. On-chip integration of electronics is particularly advantageous because of the potential to perform many signal conditioning functions in the digital domain (versus analog signal processing) as well as to achieve a reduction in system size and cost.
  • CMOS imagers as discussed above are generally known as discussed, for example, in Nixon et al., “256×256 CMOS Active Pixel Sensor Camera-on-a-Chip,” IEEE Journal of Solid-State Circuits, Vol. 31(12) pp. 2046-2050, 1996; Mendis et al, “CMOS Active Pixel Image Sensors,” IEEE Transactions on Electron Devices, Vol. 41(3) pp. 452-453, 1994 as well as U.S. Pat. No. 5,708,263, U.S. Pat. No. 5,471,515, and U.S. Pat. No. 6,204,524, which are herein incorporated by reference.
  • Recent advances in CMOS image sensor technology include the integration of the imager and sophisticated image processing modules on a single die, as mentioned above. These systems-on-a-chip (SOCs) usually have cost, power consumption and form-factor advantages over the multi-chip solutions with the same functionality. Furthermore, image processing modules integrated with CMOS imagers can be fine tuned to the specific properties of the given imager and to the needs of the targeted applications. One segment of the market where the low power of CMOS imagers is most advantageous is the mobile devices market. Many cell-phone designs are incorporating image sensors in their architectures. However, bandwidth limitations imposed by wireless transmission, together with the desire to employ image sensors with large pixel counts, necessitates the use of image compression in the system. Many of the existing designs draw on the ability of the on-board CPU to perform image compression, but at speeds far below video rates. Other available solutions rely on an additional image processing chip to perform color processing and compression.
  • All of the known solutions require a frame buffer memory to allow for rate conversion between video rate of incoming uncompressed data and the rate at which compression can be performed by either system or dedicated CPU. This leads to increased cost of the imager module and reduced video throughput of the system. Accordingly, it would be desirable to provide a CMOS imager with on-board image compression circuitry which processes image data in real time, and thus eliminates the need for a frame buffer memory.
  • SUMMARY OF THE INVENTION
  • The present invention provides a CMOS imager in which a CMOS image sensor, a color image processing module and an image compression module are all provided on a single die. Both the color image processing module and the image compression module incorporate pipelined architectures to process the image data at a video rate in a massively parallel fashion.
  • Additional advantages and features of the present invention will be apparent from the following detailed description and drawings, which illustrate preferred embodiments of the invention.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 is a block diagram of a typical CMOS pixel sensor chip.
  • FIG. 2 is a block diagram of a preferred embodiment of the present invention.
  • FIG. 3 is a block diagram of the baseline JPEG compression module of the present invention.
  • FIG. 4 is a block diagram of the SRAM Addressing Scheme of the present invention.
  • FIG. 5 is a block diagram of the FIFO & Register Control module of the present invention.
  • FIG. 6 is an illustration of a computer system having a CMOS imager according to the present invention.
  • DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
  • In the following detailed description, reference is made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized, and that structural, logical and electrical changes may be made without departing from the spirit and scope of the present invention.
  • FIG. 1 illustrates a block diagram for a typical CMOS imager 10 having a pixel array 100 with each pixel cell being constructed in the manner disclosed, for example, in U.S. Pat. No. 6,204,524, the disclosure of which is herein incorporated by reference. Pixel array 100 comprises a plurality of pixels arranged in a predetermined number of columns and rows. The pixels of each row in array 100 are all turned on at the same time by a row select line, and the pixels of each column are selectively output by a column select line. A plurality of rows and column lines are provided for the entire array 100. The row lines are selectively activated by the row driver 110 in response to row address decoder 120 and the column select lines are selectively activated by the column driver 160 in response to column address decoder 170. Thus, a row and column address is provided for each pixel. The CMOS imager is operated by the control circuit 150 which controls address decoders 120, 170 for selecting the appropriate row and column lines for pixel readout, and row and column driver circuitry 110, 160 which apply driving voltage to the drive transistors of the selected row and column lines.
  • In the operation of the FIG. 1 CMOS imager, one of the rows of pixels is selected for readout by using row driver 110 as described above. All pixels on a selected row are processed simultaneously and sampled onto a capacitor at the bottom of their respective columns. The column-parallel sampling process typically takes 1-10 mμ sec, and preferably occurs during the so-called horizontal blanking interval of a video image. Each column is then successively selected for read-out of the voltage stored in the capacitor of that column using column driver 160. The output data from CMOS imager 10 is then processed in accordance with the present invention as described below.
  • With reference to the block diagram of FIG. 2, in a preferred embodiment of the present invention, color image processor 202 converts digitized imager data (output from CMOS imager 10 of FIG. 1) from its native format (such as RGB) into a format more suitable for image compression, such as YCrCb (according to ISO CCIR-601). Once the image data is converted into appropriate for compression format, the data stream is passed to highly pipelined compression module 204. In the preferred embodiment of the present invention, compression module 204 implements JPEG compression at the video rate. In this embodiment, there are a number of line buffers 206, 208 at the front end of the compression designed to allow two-dimensional processing. In the preferred embodiment of the present invention, JPEG compression module 204 implements baseline JPEG compression, as described below in more detail with reference to FIG. 3. FIFO & Register & CPU Control module 205 implements all of the interface control and communicates with each of the modules and the outside CPU data bus.
  • FIG. 3 provides a more detailed block of baseline JPEG compression module 204. The circuitry is arranged in such a way as to pass the data from one stage to the next, allowing the entire compression operation to be performed at the video pixel rate. JPEG compression module 204 divides up the image into 8 by 8 pixel blocks in module 300, and then calculates the fast discrete cosine transform (FDCT) of each block in module 301. Quantizer module 302 rounds off the FDCT coefficients according to a specified quantization matrix. This step produces the “lossy” nature of JPEG, but allows for large compression ratios. Zigzag module 303 then performs variable length coding on these coefficients, followed by Huffman coding in module 305, which then outputs the compressed binary bitstream. For decompression, JPEG recovers the quantized DCT coefficients from the compressed data stream, takes the inverse transforms and displays the image.
  • The present invention utilizes a unique SRAM addressing scheme to reduce the memory area and improve the utilization rate of memory to maximum. Referring back to FIG. 2, the output from Color Image Processing module 202 is continually written to the corresponding SRAM 206, 208 line by line. Since JPEG compression module 204 works on 8*8 blocks of data, 8 lines of data have to be ready before compression. The challenge is how to read and write to the SRAM at the same time. There are two addressing schemes to address this problem. The first scheme uses a “ping-pong” algorithm, which uses two SRAMs, one to store the data from Color Image Processing module, and the other for JPEG compression. This scheme switches SRAM banks after every interval of 8 lines of data. The second scheme uses only one SRAM. The JPEG compression module 204 begins to read after 8 lines of data have been written to the SRAM 206, 208. Then, the output from Color Image Processing module 202 is written to the location in the SRAM from where the data has just been read. Compared with the ping-pong scheme, the latter scheme uses only one-half of the amount of SRAM. In the present invention, the second scheme is utilized, thereby greatly reducing the chip area.
  • With reference to the block diagram of FIG. 4, the detail of the SRAM addressing is shown. To reduce the chip area, the SRAM uses the minimum size to hold 8 lines of image data for JPEG compression. Module 401 automatically detects the video size and self-adjusts the addressing scheme. Module 402 delays reading for eight lines so that full 8 lines of data are ready for JPEG processing. Module 404 extracts two dimensional data from the linear SRAM, separates Y, Cb and Cr to reconstruct 8*8 blocks, and outputs the data in block order of Y Y Cb Cr, while new video data is simultaneously written to the SRAM. In each block, the data is output line by line. To provide real time video, the calculation of read and write addresses is performed “on-the-fly” without overwriting video data. All addresses including Y read address, Cr read address, Cb read address, Y write address, Cr write address, and Cb write address are calculated, pixel by pixel, which adds to the complexity of the scheme. The following formulas are used for all address calculations. The coefficients are adjusted every 8 lines. LineSize denotes the number of columns in the video frame.
    Address_next =  Clip (Address_current + Delta_current)
    Delta_next =  Delta_current * ( LineSize / 8 ) − ( LineSize − 1 ) *
    [ Delta_current * ( LineSize / 8) / ( LineSize−1 ) ]
    Clip(A) = A − LineSize −1 when A > (LineSize−1)
    = 2 * ( LineSize −1 ) hen A> 2 * ( LineSize −1)
  • Due to the nature of JPEG encoding, the output data rate of the encoder is variable (intermittent). This might present a problem for the overall system architecture, as in the absence of the frame buffer there would be a need to constantly monitor the availability of the compressed data for transfer to the system memory or direct transmission to the remote site. This type of system behavior would unnecessarily tie-up system resources (such as CPU cycles and data bus bandwidth). In order to address this problem, the present invention incorporates a relatively small memory buffer, FIFO 210, at the output of the encoder. This buffer, being much smaller that a full frame buffer, preserves the cost advantages of the system and allows for periods of fixed-rate data output interspersed with periods of inactivity.
  • In the preferred embodiment of the present invention, output buffer 210 is a dual-ported memory together with pointer control block, allowing the buffer to function as a FIFO (first-in-first-out). In this embodiment, the pointer control block allows for storage of the output of the encoder as soon as it becomes available, while allowing independent retrieval of the data based on the external requests, as long as FIFO is not empty. In yet another embodiment of this invention, the FIFO control block generates Half-Full and/or Almost Empty/Over X bytes flags describing the state of the content of the FIFO. These signals can then be used as an interrupt for external controller, prompting data retrieval cycle and relieving the external controller from the need to constantly monitor data availability in the FIFO.
  • FIG. 5 shows a more detailed block diagram of the FIFO & Register & CPU Control module, which implements the communication with the other modules. CPU Interface Control Module 502 communicates with the CPU data bus through control signals and bi-directional data bus for various operations, such as register read/write, FIFO data read, FIFO current usage, INT status, etc. Decoder module 503 handles control signal decoding. Module 506 generates an INT signal according to the current FIFO status. Reading or writing to a specified register via the Register Bus (FIG. 2) is coordinated through Register Interface Control module 500. FIFO Interface Control module 504 controls the reading process from FIFO. All interfaces are controlled and synchronized by Overall Interface Control module 501.
  • In the preferred embodiment of the present invention, the INT signal is supplied to access the internal FIFO over the system data/memory bus. To read video data correctly from the internal FIFO, the CPU must know the current state of FIFO. The following FIFO conditions will generate the INT signal transition and will also be reflected as the corresponding INT status register bits. The CPU inquiries the INT status register to get the current FIFO states. Based on the FIFO status, the CPU initiates or stops the video data read process or processes the data.
  • Bit 0: End of video frame is in FIFO
  • Bit 1: FIFO overflow
  • Bit 2: FIFO empty
  • Bit 3: End of Frame is read from FIFO
  • Bit 4: X bytes in FIFO where X is programmable
  • In a preferred embodiment of the present invention, the FIFO outputs are connected directly to the system data/memory bus, allowing for the access to the imager system to be performed over standard system communication channel in a way similar to the memory access. In yet another embodiment of the invention, the access to the registers/control functions of the imager system itself is also performed through the same bus interface. In this embodiment, the control signals are provided to allow distinguishing between various traffic over the output pins: image access, register write and register read access.
  • In addition to providing real-time compressed data stream, the imager system of the present invention may also need to provide uncompressed video stream either in full frame format or in decimated format (such as VGA image decimated to CIF resolution). Accordingly, in another embodiment of the present invention, the encoder output (compressed data stream) and the uncompressed video can be multiplexed to the input port of the output memory buffer (FIFO).
  • A typical processor based system that includes a CMOS imager device according to the present invention is illustrated generally at 600 in FIG. 6. A processor based system is exemplary of a system having digital circuits which could include CMOS imager devices. Without being limiting, such a system could include a computer system, camera system, scanner, machine vision system, vehicle navigation system, video telephone, surveillance system, auto focus system, star tracker system, motion detection system, image stabilization system and data compression system for high-definition television, all of which can utilize the present invention.
  • A processor system, such as a computer system, for example generally comprises a central processing unit (CPU) 644 that communicates with an input/output (I/O) device 646 over a bus 652. The CMOS imager 610 also communicates with the system over bus 652. The computer system 600 also includes random access memory (RAM) 648, and, in the case of a computer system may include peripheral devices such as a floppy disk drive 654 and a compact disk (CD) ROM drive 656 which also communicate with CPU 644 over the bus 652. As described above, CMOS imager 610 is combined with a pipelined JPEG compression module in a single integrated circuit.
  • The above description and drawings illustrate a preferred embodiment which achieves the objects, features and advantages of the present invention. It is not intended that the present invention be limited to the illustrated embodiments. Any modification of the present invention which comes within the spirit and scope of the following claims should be considered part of the present invention.

Claims (15)

1-25. (canceled)
26. A CMOS imaging device integrated on a single chip, comprising:
a CMOS image sensor;
a color image processing module;
an image storage module coupled to said color image processing module; and
an image compression module coupled to said image storage module,
wherein said image compression module has a pipelined architecture for processing image data in a parallel fashion at video rates without requiring a full frame memory buffer,
wherein said image compression module implements JPEG compression at video rates, and
wherein said image storage module comprises a plurality of line buffers at a front end of the compression module to allow two-dimensional processing.
27. A CMOS imaging device integrated on a single chip, comprising:
a CMOS image sensor;
a color image processing module;
an image storage module coupled to said color image processing module; and
an image compression module coupled to said image storage module,
wherein said image compression module has a pipelined architecture for processing image data in a parallel fashion at video rates without requiring a full frame memory buffer,
wherein said image compression module implements JPEG compression at video rates, and
wherein said image storage module uses an SRAM addressing scheme to allow maximum utilization rate of memory, comprising reading and writing to one SRAM at the same rate without overwriting.
28. The CMOS imaging device of claim 27, wherein an address calculation formula is used in SRAM addressing.
29. A CMOS imaging device integrated on a single chip, comprising:
a CMOS image sensor;
a color image processing module;
an image storage module coupled to said color image processing module;
an image compression module coupled to said image storage module; and
a buffer at the output of the image compression module to allow for periods of fixed-rate data output interspersed with periods of inactivity,
wherein said image compression module has a pipelined architecture for processing image data in a parallel fashion at video rates without requiring a full frame memory buffer, and
wherein said image compression module implements JPEG compression at video rates.
30. The CMOS imager device of claim 29, wherein the output buffer is a dual-ported memory provided with a pointer control block, such that the buffer functions as a FIFO memory.
31. The CMOS imager device of claim 30, wherein the pointer control block allows for the storage of the output of the encoder as soon as it becomes available, while allowing independent retrieval of data based on external requests.
32. The CMOS imager device of claim 30, further comprising a FIFO & Register & CPU control block to enable register access and FIFO access via a single system data/memory bus.
33. The CMOS imager device of claim 32, wherein the FIFO & Register & CPU control block generates an INT signal reflecting the current state of the FIFO.
34. The CMOS imager device of claim 30, wherein the FIFO pointer control block generates half-full and almost-empty/full flags reflecting the state of the content of the FIFO memory.
35. The CMOS imager device of claim 30, wherein the FIFO outputs are connected directly to a data/memory bus of a system, allowing for access to the system to be performed over standard system communication channels.
36. The CMOS imager device of claim 35, wherein access to registers and control functions of the imager system is also performed over said data/memory bus.
37. The CMOS imager device of claim 32, wherein control signals are provided to distinguish between image access, register write, and register read traffic over the output pins.
38. The CMOS imager device of claim 30, further comprising circuitry for multiplexing compressed and uncompressed video to an input port of the FIFO memory.
39. A process for processing image data from a CMOS imager device, comprising the steps of:
converting an image to image data using a CMOS image sensor;
color image processing said image data from a first format which is native to a CMOS imager to a second format more suitable for image compression;
storing said processed image data in a memory; and
compressing said stored image data using a processing module having a pipelined architecture for processing image data in a parallel fashion at video rates without requiring a full frame memory buffer,
wherein said step of compressing the image comprises JPEG compression at video rates, and
wherein said step of storing said processed image data in a memory comprises using an SRAM addressing scheme to allow maximum utilization rate of memory, said SRAM addressing scheme comprising reading and writing to one SRAM at the same rate without overwriting.
US11/524,534 2002-08-27 2006-09-21 CMOS image sensor apparatus with on-chip real-time pipelined JPEG compression module Abandoned US20070097239A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US11/524,534 US20070097239A1 (en) 2002-08-27 2006-09-21 CMOS image sensor apparatus with on-chip real-time pipelined JPEG compression module

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US10/227,809 US7532760B2 (en) 2002-08-27 2002-08-27 CMOS image sensor apparatus with on-chip real-time pipelined JPEG compression module
US11/524,534 US20070097239A1 (en) 2002-08-27 2006-09-21 CMOS image sensor apparatus with on-chip real-time pipelined JPEG compression module

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US10/227,809 Continuation US7532760B2 (en) 2002-08-27 2002-08-27 CMOS image sensor apparatus with on-chip real-time pipelined JPEG compression module

Publications (1)

Publication Number Publication Date
US20070097239A1 true US20070097239A1 (en) 2007-05-03

Family

ID=31975978

Family Applications (2)

Application Number Title Priority Date Filing Date
US10/227,809 Active 2024-07-11 US7532760B2 (en) 2002-08-27 2002-08-27 CMOS image sensor apparatus with on-chip real-time pipelined JPEG compression module
US11/524,534 Abandoned US20070097239A1 (en) 2002-08-27 2006-09-21 CMOS image sensor apparatus with on-chip real-time pipelined JPEG compression module

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US10/227,809 Active 2024-07-11 US7532760B2 (en) 2002-08-27 2002-08-27 CMOS image sensor apparatus with on-chip real-time pipelined JPEG compression module

Country Status (1)

Country Link
US (2) US7532760B2 (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8687086B1 (en) * 2012-03-30 2014-04-01 Gopro, Inc. On-chip image sensor data compression
US8855712B2 (en) 2008-08-19 2014-10-07 Digimarc Corporation Mobile phone using dedicated and programmable processors for pipelined image processing, and method thereof
US20160004680A1 (en) * 2006-09-26 2016-01-07 Sony Corporation Content management apparatus, web server, network system, content management method, content information management method, and program
US20170132750A1 (en) * 2013-07-18 2017-05-11 Imagination Technologies Limited Multi-line image processing with parallel processing units
US20190394402A1 (en) * 2018-06-20 2019-12-26 Samsung Electronics Co., Ltd. Image sensor and electronic apparatus including the same

Families Citing this family (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2005101849A1 (en) * 2004-04-09 2005-10-27 Nokia Corporation Compressed image data file creation method, image data compression device, and imaging device
US7518651B2 (en) * 2004-05-28 2009-04-14 Aptina Imaging Corporation Multiple image autofocus
JP4534630B2 (en) * 2004-07-01 2010-09-01 ソニー株式会社 IMAGING DEVICE, IMAGING ELEMENT, AND METHOD FOR CONTROLLING IMAGING DEVICE
US7495325B2 (en) * 2005-05-05 2009-02-24 Stats Chippac, Ltd. Optical die-down quad flat non-leaded package
US7496235B2 (en) 2005-08-03 2009-02-24 Micron Technology, Inc. Scan line to block re-ordering buffer for image compression
US20070041391A1 (en) * 2005-08-18 2007-02-22 Micron Technology, Inc. Method and apparatus for controlling imager output data rate
US8556165B2 (en) * 2005-09-21 2013-10-15 Bank Of America Corporation Method and system for enabling teller presentation of pre-approved credit offers
US20080007637A1 (en) * 2006-07-07 2008-01-10 Honeywell International, Inc. Image sensor that provides compressed data based on junction area
US7602427B2 (en) * 2006-07-07 2009-10-13 Honeywell International Inc. Image sensor that provides raw output comprising compressed data
US20080007550A1 (en) * 2006-07-07 2008-01-10 Honeywell International, Inc. Current driven display for displaying compressed video
US7920086B2 (en) * 2006-07-07 2011-04-05 Honeywell International Inc. Display for displaying compressed video
US20080018624A1 (en) * 2006-07-07 2008-01-24 Honeywell International, Inc. Display for displaying compressed video based on sub-division area
US8462377B2 (en) * 2007-07-25 2013-06-11 Aptina Imaging Corporation Method, apparatus, and system for reduction of line processing memory size used in image processing
GB2461912A (en) * 2008-07-17 2010-01-20 Micron Technology Inc Method and apparatus for dewarping and/or perspective correction of an image
TWI532378B (en) 2010-09-09 2016-05-01 晨星半導體股份有限公司 Image encoding data temporary storage device and the image encoding data temporary storage method thereof
AU2013295553B2 (en) 2012-07-26 2017-10-19 DePuy Synthes Products, Inc. Continuous video in a light deficient environment
IN2015MN00022A (en) 2012-07-26 2015-10-16 Olive Medical Corp
JP6363114B2 (en) 2013-02-28 2018-07-25 デピュイ・シンセス・プロダクツ・インコーポレイテッド Video strobe copy of vocal cords with CMOS sensor
CA2906798A1 (en) 2013-03-15 2014-09-18 Olive Medical Corporation Super resolution and color motion artifact correction in a pulsed color imaging system
US10251530B2 (en) 2013-03-15 2019-04-09 DePuy Synthes Products, Inc. Scope sensing in a light controlled environment
JP6404318B2 (en) 2013-03-15 2018-10-10 デピュイ・シンセス・プロダクツ・インコーポレイテッド Integrated optical energy control of laser pulses
US10084944B2 (en) 2014-03-21 2018-09-25 DePuy Synthes Products, Inc. Card edge connector for an imaging sensor
JP6355595B2 (en) * 2015-06-02 2018-07-11 キヤノン株式会社 IMAGING ELEMENT, IMAGING DEVICE, IMAGING ELEMENT CONTROL METHOD, PROGRAM, AND STORAGE MEDIUM
CN106931964B (en) * 2017-01-19 2019-12-03 中国人民解放军国防科学技术大学 Attitude determination method and star sensor based on compressed sensing imaging
DE102017108016A1 (en) * 2017-04-13 2018-10-18 Carl Zeiss Microscopy Gmbh Microscope system and method for operating a microscope system
CN109032125B (en) * 2018-05-31 2021-09-10 上海工程技术大学 Navigation method of visual AGV

Citations (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4237494A (en) * 1978-07-20 1980-12-02 Ricoh Company, Ltd. Facsimile system
US4833651A (en) * 1986-07-24 1989-05-23 National Semiconductor Corporation High-speed, asynchronous, No-Fall-Through, first-in-first out memory with high data integrity
US5084841A (en) * 1989-08-14 1992-01-28 Texas Instruments Incorporated Programmable status flag generator FIFO using gray code
US5307458A (en) * 1991-12-23 1994-04-26 Xerox Corporation Input/output coprocessor for printing machine
US5390350A (en) * 1991-04-22 1995-02-14 Western Digital Corporation Integrated circuit chip core logic system controller with power saving features for a microcomputer system
US5406554A (en) * 1993-10-05 1995-04-11 Music Semiconductors, Corp. Synchronous FIFO having an alterable buffer store
US5434593A (en) * 1992-10-13 1995-07-18 Gould Instrument Systems, Inc. Display resolution enhancement using data compression and overlapping techniques
US5450599A (en) * 1992-06-04 1995-09-12 International Business Machines Corporation Sequential pipelined processing for the compression and decompression of image data
US5471515A (en) * 1994-01-28 1995-11-28 California Institute Of Technology Active pixel sensor with intra-pixel charge transfer
US5659362A (en) * 1994-09-07 1997-08-19 University Of South Florida VLSI circuit structure for implementing JPEG image compression standard
US5708263A (en) * 1995-12-27 1998-01-13 International Business Machines Corporation Photodetector array
US5835691A (en) * 1995-10-20 1998-11-10 Fuji Xerox Co., Ltd. Image processing apparatus with detection of an overflow and underflow in a code buffer
US5903261A (en) * 1996-06-20 1999-05-11 Data Translation, Inc. Computer based video system
US6192492B1 (en) * 1995-06-19 2001-02-20 Seagate Technology Llc Fast ATA-compatible drive interface with error detection and/or error correction
US6204524B1 (en) * 1999-07-14 2001-03-20 Micron Technology, Inc. CMOS imager with storage capacitor
US20010040635A1 (en) * 1994-01-28 2001-11-15 California Institute Of Technology CMOS active pixel sensor type imaging system on a chip
US6452632B1 (en) * 1997-01-31 2002-09-17 Kabushiki Kaisha Toshiba Solid state image sensor and video system using the same
US20030160883A1 (en) * 2000-09-12 2003-08-28 Viktor Ariel Single chip cmos image sensor system with video compression
US6757019B1 (en) * 1999-03-13 2004-06-29 The Board Of Trustees Of The Leland Stanford Junior University Low-power parallel processor and imager having peripheral control circuitry
US6920510B2 (en) * 2002-06-05 2005-07-19 Lsi Logic Corporation Time sharing a single port memory among a plurality of ports
US6927794B2 (en) * 1999-12-09 2005-08-09 Minolta Co., Ltd. Digital camera provided with a plurality of mounting portions to each of which a recording medium is mounted
US6984205B2 (en) * 1999-03-01 2006-01-10 Gazdzinski Robert F Endoscopic smart probe and method

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5165103A (en) * 1986-09-12 1992-11-17 Hitachi, Ltd. Image data processing method for selective partial image storage
US5300724A (en) * 1989-07-28 1994-04-05 Mark Medovich Real time programmable, time variant synthesizer
US5298997A (en) * 1991-10-01 1994-03-29 Ricoh Company, Ltd. Line data-block data converting circuit
US5668648A (en) * 1991-11-26 1997-09-16 Kabushiki Kaisha Toshiba Computer-assisted holographic display apparatus
US5649083A (en) * 1994-04-15 1997-07-15 Hewlett-Packard Company System and method for dithering and quantizing image data to optimize visual quality of a color recovered image
US5774754A (en) * 1994-04-26 1998-06-30 Minolta Co., Ltd. Camera capable of previewing a photographed image
US5671440A (en) * 1994-08-08 1997-09-23 Eastman Kodak Company Color image data reorientation and format conversion system
US5778218A (en) * 1996-12-19 1998-07-07 Advanced Micro Devices, Inc. Method and apparatus for clock synchronization across an isochronous bus by adjustment of frame clock rates
US6078532A (en) * 1999-02-01 2000-06-20 Cisco Technology Inc. Method and apparatus for improving performance of DRAM subsystems with SRAM overlays
US6701390B2 (en) * 2001-06-06 2004-03-02 Koninklijke Philips Electronics N.V. FIFO buffer that can read and/or write multiple and/or selectable number of data words per bus cycle

Patent Citations (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4237494A (en) * 1978-07-20 1980-12-02 Ricoh Company, Ltd. Facsimile system
US4833651A (en) * 1986-07-24 1989-05-23 National Semiconductor Corporation High-speed, asynchronous, No-Fall-Through, first-in-first out memory with high data integrity
US5084841A (en) * 1989-08-14 1992-01-28 Texas Instruments Incorporated Programmable status flag generator FIFO using gray code
US5390350A (en) * 1991-04-22 1995-02-14 Western Digital Corporation Integrated circuit chip core logic system controller with power saving features for a microcomputer system
US5307458A (en) * 1991-12-23 1994-04-26 Xerox Corporation Input/output coprocessor for printing machine
US5450599A (en) * 1992-06-04 1995-09-12 International Business Machines Corporation Sequential pipelined processing for the compression and decompression of image data
US5434593A (en) * 1992-10-13 1995-07-18 Gould Instrument Systems, Inc. Display resolution enhancement using data compression and overlapping techniques
US5406554A (en) * 1993-10-05 1995-04-11 Music Semiconductors, Corp. Synchronous FIFO having an alterable buffer store
US20010040635A1 (en) * 1994-01-28 2001-11-15 California Institute Of Technology CMOS active pixel sensor type imaging system on a chip
US5471515A (en) * 1994-01-28 1995-11-28 California Institute Of Technology Active pixel sensor with intra-pixel charge transfer
US5659362A (en) * 1994-09-07 1997-08-19 University Of South Florida VLSI circuit structure for implementing JPEG image compression standard
US6192492B1 (en) * 1995-06-19 2001-02-20 Seagate Technology Llc Fast ATA-compatible drive interface with error detection and/or error correction
US5835691A (en) * 1995-10-20 1998-11-10 Fuji Xerox Co., Ltd. Image processing apparatus with detection of an overflow and underflow in a code buffer
US5708263A (en) * 1995-12-27 1998-01-13 International Business Machines Corporation Photodetector array
US5903261A (en) * 1996-06-20 1999-05-11 Data Translation, Inc. Computer based video system
US6452632B1 (en) * 1997-01-31 2002-09-17 Kabushiki Kaisha Toshiba Solid state image sensor and video system using the same
US6984205B2 (en) * 1999-03-01 2006-01-10 Gazdzinski Robert F Endoscopic smart probe and method
US6757019B1 (en) * 1999-03-13 2004-06-29 The Board Of Trustees Of The Leland Stanford Junior University Low-power parallel processor and imager having peripheral control circuitry
US6204524B1 (en) * 1999-07-14 2001-03-20 Micron Technology, Inc. CMOS imager with storage capacitor
US6927794B2 (en) * 1999-12-09 2005-08-09 Minolta Co., Ltd. Digital camera provided with a plurality of mounting portions to each of which a recording medium is mounted
US20030160883A1 (en) * 2000-09-12 2003-08-28 Viktor Ariel Single chip cmos image sensor system with video compression
US6920510B2 (en) * 2002-06-05 2005-07-19 Lsi Logic Corporation Time sharing a single port memory among a plurality of ports

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160004680A1 (en) * 2006-09-26 2016-01-07 Sony Corporation Content management apparatus, web server, network system, content management method, content information management method, and program
US10650090B2 (en) * 2006-09-26 2020-05-12 Sony Corporation Content management apparatus, web server, network system, content management method, content information management method, and program
US8855712B2 (en) 2008-08-19 2014-10-07 Digimarc Corporation Mobile phone using dedicated and programmable processors for pipelined image processing, and method thereof
US9338373B2 (en) 2012-03-30 2016-05-10 Gopro, Inc. Image sensor data compression and DSP decompression
US8970726B2 (en) * 2012-03-30 2015-03-03 Gopro, Inc. Image sensor data compression and DSP decompression
US9253421B2 (en) * 2012-03-30 2016-02-02 Gopro, Inc. On-chip image sensor data compression
US8687086B1 (en) * 2012-03-30 2014-04-01 Gopro, Inc. On-chip image sensor data compression
US10044899B2 (en) 2012-03-30 2018-08-07 Gopro, Inc. On-chip image sensor data compression
US10326904B2 (en) 2012-03-30 2019-06-18 Gopro, Inc. On-chip image sensor data compression
US20140152872A1 (en) * 2012-03-30 2014-06-05 Gopro, Inc. On-chip image sensor data compression
US10701291B2 (en) 2012-03-30 2020-06-30 Gopro, Inc. On-chip image sensor data compression
US11375139B2 (en) 2012-03-30 2022-06-28 Gopro, Inc. On-chip image sensor data compression
US20170132750A1 (en) * 2013-07-18 2017-05-11 Imagination Technologies Limited Multi-line image processing with parallel processing units
US9779470B2 (en) * 2013-07-18 2017-10-03 Imagination Technologies Limited Multi-line image processing with parallel processing units
US20190394402A1 (en) * 2018-06-20 2019-12-26 Samsung Electronics Co., Ltd. Image sensor and electronic apparatus including the same

Also Published As

Publication number Publication date
US20040042668A1 (en) 2004-03-04
US7532760B2 (en) 2009-05-12

Similar Documents

Publication Publication Date Title
US7532760B2 (en) CMOS image sensor apparatus with on-chip real-time pipelined JPEG compression module
USRE44888E1 (en) Solid state image sensor and video system using the same
US10277843B2 (en) Oversampled image sensor with conditional pixel readout
Funatsu et al. 6.2 133Mpixel 60fps CMOS image sensor with 32-column shared high-speed column-parallel SAR ADCs
JP4955079B2 (en) Image sensor with interleaved image output
US7667752B2 (en) Semiconductor imaging sensor array devices with dual-port digital readout
US8514310B2 (en) Increasing readout speed in CMOS APS sensors through group readout
US20010040631A1 (en) CMOS sensor array with a memory interface
US20040233324A1 (en) Systems and methods for power conservation in a CMOS imager
US9584696B2 (en) Imaging systems with embedded data transmission capabilities
JPH11126895A (en) Active pixel image sensor having shared amplifier readout
JPH10224696A (en) Solid-state image pickup element and image system using the solid-state image pickup element
US7936375B2 (en) Image processor, imaging device, and image processing system use with image memory
US6786411B2 (en) Image sensor array readout for simplified image compression
US7190397B2 (en) CMOS imager decoder structure
US7747090B2 (en) Scan line to block re-ordering buffer for image compression
US20010030705A1 (en) Solid-state imaging element, image processor and image processing method
Milirud et al. A wide dynamic range CMOS active pixel sensor with frame difference
JP2000236481A (en) Image pickup device and method for detecting image
RU2000129202A (en) VIDEO CAMERA
Li et al. Throughput-quality tradeoff in tethered video camera design
Lake Multiple Output Photodiode Arrays

Legal Events

Date Code Title Description
STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION