US8996177B2 - Robotic training apparatus and methods - Google Patents
Robotic training apparatus and methods Download PDFInfo
- Publication number
- US8996177B2 US8996177B2 US13/841,980 US201313841980A US8996177B2 US 8996177 B2 US8996177 B2 US 8996177B2 US 201313841980 A US201313841980 A US 201313841980A US 8996177 B2 US8996177 B2 US 8996177B2
- Authority
- US
- United States
- Prior art keywords
- training
- robotic
- robotic apparatus
- value
- cost function
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1628—Programme controls characterised by the control loop
- B25J9/163—Programme controls characterised by the control loop learning, adaptive, model based, rule based expert control
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/004—Artificial life, i.e. computing arrangements simulating life
- G06N3/008—Artificial life, i.e. computing arrangements simulating life based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S901/00—Robots
- Y10S901/02—Arm motion controller
- Y10S901/03—Teaching system
Definitions
- the present disclosure relates to robotic learning and training of robotic devices.
- Robotic control systems exist. Typically, these systems may comprise a controller apparatus configured to control a plant.
- the plant may include, for example, a robotic arm.
- the controller may receive an input signal, and may generate a control signal.
- the plant may provide a feedback signal to the controller.
- the controller may receive one or more physical inputs, which may individually comprise continuous (e.g., analog) and/or discrete (e.g., digital) signals describing various variables. Examples of such various variables may include one or more of temperature, voltage, current, orientation, position, plant state, and/or other signals.
- Programming the controller to enable a robot to perform complex functions may be not straightforward.
- the programming may become a daunting task, particularly when the environment and/or characteristics of the robot (e.g., replaceable limb) may vary between the time of programming and the time of operation.
- the method may comprise: establishing a target trajectory for an operation of the robotic apparatus, the target trajectory being characterized by a performance parameter configured based on a cost function; adjusting, with one or more processors, instantaneous cost value based on a training signal, the training signal being provided based on an evaluation of a measure between a current value of the performance parameter and a target value of the performance parameter, the robotic apparatus being configured to receive the training signal via a remote link; and determining, with one or more processors, an updated cost function based on the adjusted instantaneous cost value and the cost function, the updated cost function being configured to transition the performance parameter towards the target value.
- the robotic apparatus may be configured to operate autonomously subsequent receipt of the training signal.
- the robotic apparatus may be configured to receive the training signal via the remote link based on the evaluation being performed by a human operator.
- the evaluation may be based on visual observation of operation of the robotic apparatus.
- the evaluation may be devoid of computations of the cost function.
- the robotic apparatus may comprise at least one artificial limb.
- the performance parameter may be configured based on a function of acceleration of at least a portion of the limb.
- the evaluation may be based on a comparison between (i) a first trajectory motion of the limb during performance of a first task by the robotic apparatus and (ii) a second trajectory motion of a limb of a human during performance of the first task by the human.
- the function of the acceleration may comprise jerk.
- the jerk may be determined based on a temporal derivative of the acceleration.
- the limb may comprise an artificial leg and the task comprises a gating motion towards an object.
- the limb may comprise an artificial hand and the task comprises grasping of an object by the artificial hand.
- the limb may comprise an artificial arm.
- the artificial hand may be disposed at a distal end of the artificial arm.
- the transition of the performance parameter towards the target value may be based on a minimization of the cost function.
- the training signal may be configured to increase the instantaneous cost value based on the current performance value being separated from the target performance value by a first margin.
- the training signal may be configured to decrease the instantaneous cost value based on the current performance value being within the target performance value by a second margin.
- the training signal may comprise two or more levels, individual ones of the two or more levels configured based on a distance measure between the current performance and the target performance.
- the adjustment of the instantaneous cost value may be configured based on a one of the two or more levels.
- the target trajectory may be characterized by at least one state parameter.
- the cost function may be configured based on the state parameter.
- the instantaneous cost value may be configured to adjust the cost function within a range of the state parameter values.
- adjusting the cost function may comprise one or more of a concave, a convex, or a step function determined for the range of the state parameter values.
- the state parameter may comprise one or both of a spatial coordinate or a motion characteristic associated with the robotic apparatus.
- the instructions may be executable by a processor to perform a method for training a computerized robotic apparatus.
- the method may comprise: establishing a target trajectory for an operation of the robotic apparatus, the target trajectory being characterized by a performance parameter configured based on a cost function; adjusting, with one or more processors, instantaneous cost value based on a training signal, the training signal being provided based on an evaluation of a measure between a current value of the performance parameter and a target value of the performance parameter, the robotic apparatus being configured to receive the training signal via a remote link; and determining, with one or more processors, an updated cost function based on the adjusted instantaneous cost value and the cost function, the updated cost function being configured to transition the performance parameter towards the target value.
- the system may comprise a training apparatus and a robotic apparatus.
- the training apparatus may be configured to train the robotic apparatus.
- the robotic apparatus may comprise one or more processors configured to execute computer program modules.
- the computer program modules may comprise a first logic module, a second logic module, and a third logic module.
- the first logic module may be configured to receive a training signal from the training apparatus.
- the training signal may be provided based on an evaluation of a measure between a current value of a performance parameter and a target value of the performance parameter.
- the second logic module may be configured to operate the robotic apparatus in accordance with a target trajectory characterized by the performance parameter.
- the third logic module may be configured to (1) adjust an instantaneous cost value and (2) determine an updated cost function configured to transition the performance parameter towards the target value, the updated cost function being based on an initial cost function associated with the performance parameter and the adjusted instantaneous cost value.
- the training apparatus may comprise a remote evaluation apparatus configured to communicate with the robotic apparatus via one or more of radio waves, sound waves, infrared waves, or visible light waves.
- the remote evaluation apparatus may comprise a first button and a second button.
- the first button may be configured to facilitate communication of positive reinforcement to the robotic apparatus based on performance associated with the performance parameter being within a given range.
- the second button may be configured to facilitate communication of negative reinforcement to the robotic apparatus based on the performance being outside the given range.
- the system may comprise a user interface apparatus configured to present information associated with a training history.
- the training history may comprise information associated with (1) timing of one or more events associated with receipt of one or more training signals and (2) magnitude of adjustment of the instantaneous cost associated with individual ones of the one or more events.
- the target trajectory may be characterized by at least one state parameter.
- the user interface apparatus may be configured to enable modification of the training history.
- the modification may comprise one or more of (1) addition of another event associated with another training signal, (2) removal of at least one existing event, or (3) adjustment of a value of the state parameter associated with the event.
- the state parameter may comprise a motion characteristic of the robotic apparatus.
- the history modification may comprise a transformation of the instantaneous cost associated with a given event into the motion characteristic.
- FIG. 1 is a block diagram illustrating a generalized adaptive robotic apparatus for use with the training methodology, in accordance with one or more implementations.
- FIG. 2 is a block diagram illustrating an adaptive robotic apparatus comprising a cost function constructor for use with the training methodology based on reinforcement, in accordance with one or more implementations.
- FIG. 3A is a plot illustrating cost function versus state during training of an adaptive controller comprising positive reinforcement, in accordance with some implementations.
- FIG. 3B is a plot illustrating cost function versus state during training of an adaptive controller comprising negative reinforcement, in accordance with some implementations.
- FIG. 3C is a plot illustrating cost function versus state during training of an adaptive controller comprising Gaussian cost adjustment due to reinforcement, in accordance with some implementations.
- FIG. 4 is a plot illustrating cost function versus time corresponding to a training trajectory of a robotic device comprising positive and negative reinforcement, in accordance with some implementations.
- FIG. 5 is a graphical illustration depicting exemplary trajectories of a robotic device during training, in accordance with some implementations.
- FIG. 6 is a block diagram illustrating a clicker apparatus for controller system, in accordance with one or more implementations.
- FIG. 7 is a plot illustrating movement of a robotic apparatus configured for use with the cost function methodology of the disclosure, in accordance with one or more implementations.
- FIG. 8 is a logical flow diagram illustrating use of reinforcement learning configured to adjust cost function of an adaptive controller, in accordance with one or more implementations.
- FIG. 9 is a logical flow diagram illustrating training of a robotic apparatus, in accordance with one or more implementations.
- FIG. 10 is a block diagram illustrating computerized system useful for efficient connection plasticity update mechanism in a spiking network, in accordance with one or more implementations.
- FIG. 11A is a block diagram illustrating a neuromorphic computerized system useful with efficient connection plasticity update mechanism in a spiking network, in accordance with one or more implementations.
- FIG. 11B is a block diagram illustrating a hierarchical neuromorphic computerized system architecture useful with efficient connection plasticity update mechanism in a spiking network, in accordance with one or more implementations.
- FIG. 11C is a block diagram illustrating cell-type neuromorphic computerized system architecture useful with efficient connection plasticity update mechanism in a spiking network, in accordance with one or more implementations.
- bus is meant generally to denote all types of interconnection or communication architecture that is used to access the synaptic and neuron memory.
- the “bus” may be optical, wireless, infrared, and/or another type of communication medium.
- the exact topology of the bus could be for example standard “bus”, hierarchical bus, network-on-chip, address-event-representation (AER) connection, and/or other type of communication topology used for accessing, e.g., different memories in pulse-based system.
- the terms “computer”, “computing device”, and “computerized device” may include one or more of personal computers (PCs) and/or minicomputers (e.g., desktop, laptop, and/or other PCs), mainframe computers, workstations, servers, personal digital assistants (PDAs), handheld computers, embedded computers, programmable logic devices, personal communicators, tablet computers, portable navigation aids, J2ME equipped devices, cellular telephones, smart phones, personal integrated communication and/or entertainment devices, and/or any other device capable of executing a set of instructions and processing an incoming data signal.
- PCs personal computers
- minicomputers e.g., desktop, laptop, and/or other PCs
- mainframe computers workstations
- servers personal digital assistants
- handheld computers handheld computers
- embedded computers embedded computers
- programmable logic devices personal communicators
- tablet computers tablet computers
- portable navigation aids J2ME equipped devices
- J2ME equipped devices J2ME equipped devices
- cellular telephones cellular telephones
- smart phones personal integrated communication and
- ⁇ may include any sequence of human and/or machine cognizable steps which perform a function.
- Such program may be rendered in a programming language and/or environment including one or more of C/C++, C#, Fortran, COBOL, MATLABTM, PASCAL, Python, assembly language, markup languages (e.g., HTML, SGML, XML, VoXML), object-oriented environments (e.g., Common Object Request Broker Architecture (CORBA)), JavaTM (e.g., J2ME, Java Beans), Binary Runtime Environment (e.g., BREW), and/or other programming languages and/or environments.
- CORBA Common Object Request Broker Architecture
- JavaTM e.g., J2ME, Java Beans
- Binary Runtime Environment e.g., BREW
- connection may include a causal link between any two or more entities (whether physical or logical/virtual), which may enable information exchange between the entities.
- memory may include an integrated circuit and/or other storage device adapted for storing digital data.
- memory may include one or more of ROM, PROM, EEPROM, DRAM, Mobile DRAM, SDRAM, DDR/2 SDRAM, EDO/FPMS, RLDRAM, SRAM, “flash” memory (e.g., NAND/NOR), memristor memory, PSRAM, and/or other types of memory.
- integrated circuit As used herein, the terms “integrated circuit”, “chip”, and “IC” are meant to refer to an electronic circuit manufactured by the patterned diffusion of trace elements into the surface of a thin substrate of semiconductor material.
- integrated circuits may include field programmable gate arrays (e.g., FPGAs), a programmable logic device (PLD), reconfigurable computer fabrics (RCFs), application-specific integrated circuits (ASICs), and/or other types of integrated circuits.
- FPGAs field programmable gate arrays
- PLD programmable logic device
- RCFs reconfigurable computer fabrics
- ASICs application-specific integrated circuits
- microprocessor and “digital processor” are meant generally to include digital processing devices.
- digital processing devices may include one or more of digital signal processors (DSPs), reduced instruction set computers (RISC), general-purpose (CISC) processors, microprocessors, gate arrays (e.g., field programmable gate arrays (FPGAs)), PLDs, reconfigurable computer fabrics (RCFs), array processors, secure microprocessors, application-specific integrated circuits (ASICs), and/or other digital processing devices.
- DSPs digital signal processors
- RISC reduced instruction set computers
- CISC general-purpose
- microprocessors gate arrays (e.g., field programmable gate arrays (FPGAs)), PLDs, reconfigurable computer fabrics (RCFs), array processors, secure microprocessors, application-specific integrated circuits (ASICs), and/or other digital processing devices.
- FPGAs field programmable gate arrays
- RCFs reconfigurable computer fabrics
- ASICs application-specific integrated
- a network interface refers to any signal, data, and/or software interface with a component, network, and/or process.
- a network interface may include one or more of FireWire (e.g., FW400, FW800, etc.), USB (e.g., USB2), Ethernet (e.g., 10/100, 10/100/1000 (Gigabit Ethernet), 10-Gig-E, etc.), MoCA, Coaxsys (e.g., TVnetTM), radio frequency tuner (e.g., in-band or OOB, cable modem, etc.), Wi-Fi (802.11), WiMAX (802.16), PAN (e.g., 802.15), cellular (e.g., 3G, LTE/LTE-A/TD-LTE, GSM, etc.), IrDA families, and/or other network interfaces.
- FireWire e.g., FW400, FW800, etc.
- USB e.g., USB2
- Ethernet e.g.,
- neurode As used herein, the terms “node”, “neuron”, and “neuronal node” are meant to refer, without limitation, to a network unit (e.g., a spiking neuron and a set of synapses configured to provide input signals to the neuron) having parameters that are subject to adaptation in accordance with a model.
- a network unit e.g., a spiking neuron and a set of synapses configured to provide input signals to the neuron having parameters that are subject to adaptation in accordance with a model.
- state and “node state” is meant generally to denote a full (or partial) set of dynamic variables used to describe node state.
- connection As used herein, the term “synaptic channel”, “connection”, “link”, “transmission channel”, “delay line”, and “communications channel” include a link between any two or more entities (whether physical (wired or wireless), or logical/virtual) which enables information exchange between the entities, and may be characterized by a one or more variables affecting the information exchange.
- Wi-Fi includes one or more of IEEE-Std. 802.11, variants of IEEE-Std. 802.11, standards related to IEEE-Std. 802.11 (e.g., 802.11 a/b/g/n/s/v), and/or other wireless standards.
- wireless means any wireless signal, data, communication, and/or other wireless interface.
- a wireless interface may include one or more of Wi-Fi, Bluetooth, 3G (3GPP/3GPP2), HSDPA/HSUPA, TDMA, CDMA (e.g., IS-95A, WCDMA, etc.), FHSS, DSSS, GSM, PAN/802.15, WiMAX (802.16), 802.20, narrowband/FDMA, OFDM, PCS/DCS, LTE/LTE-A/TD-LTE, analog cellular, CDPD, satellite systems, millimeter wave or microwave systems, acoustic, infrared (i.e., IrDA), and/or other wireless interfaces.
- an object of the training is to enable the robotic device to move like an animal.
- FIG. 1 illustrates one exemplary adaptive control system useful to the disclosure.
- the system 100 shown in FIG. 1 may comprise an adaptive controller apparatus 102 , a plant 104 , a sensor block 106 and/or other components.
- the plant 104 may comprise an unmanned robotic vehicle, drone, and/or other plants.
- the controller apparatus 102 may be configured to receive sensory input from the sensor block 106 and a feedback 108 from the plant 104 .
- the feedback signal may comprise one or more of a control plane position, motor speed and/or current, remaining battery level, status of feedback motors and/or actuators, and/or other parameters (e.g., motor commands actually sent to the plant).
- the sensory signal 112 and the feedback 108 may be referred to as the sensory input, as they describe inputs to the control process.
- the input signal may comprise data x(t) used for solving a particular control task.
- the signal x(t) may comprise a stream of sensor data, generated, for example, by the sensor block 106 in FIG. 1 .
- the sensor data may comprise a raw sensor stream. Examples of the raw sensor stream may include data relating to one or more of proximity, inertial, terrain imaging, and/or other data.
- the sensor data may comprise preprocessed data. Examples of preprocessed data may include one or more of velocity, extracted from accelerometers, distance to obstacle, positions, and/or other preprocessed data.
- the signal x(t) may comprise an array of pixel values (e.g., RGB, CMYK, HSV, HSL, grayscale, and/or other pixel values) in the input image, and/or preprocessed data (e.g., levels of activations of Gabor filters for face recognition, contours, optic flow and/or other preprocessed data).
- the input signal x(t) may comprise desired motion trajectory, for example, in order to predict future state of the robot on the basis of current state and desired motion.
- the controller apparatus 102 may be configured to provide control output 118 (e.g., motor commands) to the plant.
- the controller may be operable in accordance with a learning process configured to perform one or more tasks (e.g., locate a target, gather all pieces of garbage, and/or other tasks).
- the learning process may be configured based on a control policy that may be used to produce control output 118 .
- the policy may comprise one or more criteria such as lowest energy, fastest time, smallest instantaneous error, maximum speed within bounds, and/or other parameters.
- the policy may be based on an objective function.
- a feasible solution that minimizes (or maximizes, if that is the goal) the objective function may be referred to as an optimal solution.
- the objective function F may be referred to as the cost function (minimization), cost to go, indirect utility function (minimization), utility function (maximization), energy function, energy functional, and/or other terms relating to the objective function F.
- the cost function F may be shaped by one or more instantaneous cost C inputs associated with the operation of the robotic device.
- instantaneous cost may be determined based on a reinforcement input 110 provided by an external agent during training of the adaptive systems 100 , 220 .
- the external agent may comprise a human and/or computerized operator communicating supervisory signal via, for example, remote commands.
- the commands may be effectuated based on, for example, robotic device approaching a target, an area of danger (an obstacle), device reaching of a pre-configured condition (e.g., a safety constraint of speed in excess of 5 m/s), and/or other criteria.
- FIG. 2 illustrates the use of reinforcement input to form cost function during training of an adaptive system 220 .
- the system 220 may comprise a controller 232 configured to generate control output for plant 234 based on sensory input 228 —2 from sensor block 236 and/or plant feedback 238 —2 .
- the system 100 may be operable in accordance with a cost function f, depicted by the cost function block 226 in FIG. 2 .
- the cost function 226 may be utilized by controller constructor block 230 adapted to configure a realization of the controller 232 and form a control policy.
- the controller construct block may utilize the cost function by computing its gradient and setting the control policy in accordance with the gradient of the cost function.
- the controller construct block may use the cost function to update parameters of the controller.
- the controller construct block may execute one (or more) trials using different parameters of the controller over a trajectory. Based on values for the cost function obtained during these, a determination may be made as to which control parameters should be utilized during controller operation.
- the controller realization may be comprise a reinforcement learning process effectuated by spiking neural networks (e.g., as described in co-pending and co-owned U.S. patent application Ser. No. 13/489,280 entitled “STOCHASTIC SPIKING NETWORK APPARATUS AND METHOD”, filed Jun. 5, 2012, incorporated herein by reference in its entirety):
- the reinforcement signal 210 may include positive and/or negative reinforcement signals and/or other types of reinforcement and/or reward signals.
- the reinforcement signal may comprise sensor input from the environment (e.g., a perimeter beacon alarm and/or other input) and/or a sensor on the robotic device (e.g., a proximity alarm and/or other input).
- the reinforcement signal 210 may provide an input for the adaptive system 220
- the reinforcement signal 210 may be distinguished from the sensory inputs (e.g., the inputs 112 , 108 in FIG. 1 and/or the inputs 238 , 228 in FIG. 2 ) based on their use.
- the reinforcement signal 210 may be utilized by the control system (e.g., 100 , 220 ) in order to determine the cost function F (e.g., by the cost constructor 222 , in the implementation shown in FIG. 2 ). Once the cost function F and the controller realization is constructed (e.g., by the controller constructor 230 in FIG. 2 ), the reinforcement signal may not directly affect the operation of the controller 232 .
- the sensory inputs 112 , 108 in FIG. 1 and/or the inputs 238 , 228 in FIG. 2 may affect both: (i) the cost function 226 construction and (ii) the controller 232 operation.
- the sensory inputs may be utilized by the adaptive controller to update the control process through the eligibility trace (e.g., the eligibility trace e(t) term in Eqn. 1).
- the cost function may be constructed based on one or more events associated with the reinforcement signal (e.g., 210 in FIG. 2 ).
- the agent providing the reinforcement signal may comprise a human operator communicating with the system 100 , 220 .
- the communication may be effectuating by audio signal (e.g., a clap), video signal (e.g., a card of a certain color), and a wireless communications device (e.g., radio, infrared) hereinafter referred to as the “clicker”.
- the communication may be effectuated by a light beam of a remote device, as described in detail in co-pending and co-owned U.S.
- FIG. 6 illustrates exemplary implementations of clicker apparatus 600 , 610 useful to for communicating information to an adaptive system (e.g., the systems 100 , 220 of FIGS. 1-2 ) during training.
- the adaptive system may comprise a robotic apparatus (e.g., a robot).
- the adaptive system may comprise a computer apparatus and/or a machinery to be trained.
- a computer apparatus may include training the operating system of the computer to perform some desired functions or avoiding others; it may be training one or multiple application, alone or to work together.
- a machinery may consist of an exoskeleton configured to help walking, or carrying, lifting heavy loads.
- the adaptive controller may include an adaptive controller for a computer numerical control (CNC) industrial machine configured to learn machining a part in a particular way without the use of programming.
- the adaptive controller may be realized on a server configured to implement a virtual environment, such as a virtual world.
- the clicker apparatus 600 may comprise one or more buttons 602 , 603 configured to provide reinforcement indication.
- individual buttons 602 , 603 may be configured to provide positive and negative reinforcement signals (e.g., “good”, “bad”), respectively.
- individual buttons 602 , 603 may be configured to provide different degrees or ranges of positive and/or negative reinforcement signals (e.g., “excellent”, “good”, “bad”, and “very bad”; a number between 1 and 5; and/or other degrees or ranges).
- the clicker apparatus 600 may comprise two pairs of buttons 612 . Two individual buttons 612 of FIG. 6 may be configured to provide different degrees of positive reinforcement (e.g., “excellent”, “good”). The other two individual buttons 612 of FIG. 6 may be configured to provide different degrees of negative (e.g., “bad”, “very bad”).
- the clicker apparatus 600 , 610 may comprise a wireless communications interface 604 , 614 .
- the interface 604 may comprise one or more of a radio frequency interface, a sound wave interface, an infrared interface, and/or other interfaces.
- the interface 614 may comprise an electromagnetic wave (e.g., visual light) interface.
- FIGS. 3A-5 illustrate the use of a clicker device during training of an adaptive system (e.g., 220 of FIG. 2 ) in order to build a cost function F(q,t) (e.g., 226 ) that may be utilized by the controller constructor in order to improve performance of the adaptive system.
- performance improvement may be determined based on a decrease in the cost function over the same and/or similar trajectory or during execution of the task(s).
- the system 220 may comprise a robotic device comprising one or more artificial limbs (e.g., legs, manipulators, and/or other limbs).
- the cost function F(q,t) may be used in order to smooth out movements of the robot and/or to make the movements to appear more natural looking and/or to make them more “human-like” or more “animal-like”.
- FIG. 5 depicts exemplary trajectories of an adaptive apparatus useful in illustrating principles of the cost function design methodology of the disclosure.
- the cost function F(q,t) may be configured as the cost-to-go.
- the cost-to-go may be described as a cumulative of immediate costs C i along a trajectory (e.g., the trajectory 512 , 514 in FIG. 5 ) of the robotic device and/or when following an action policy, from a start or intermediate point to a finish point.
- the trajectory may be associated with an environment (e.g., denoted by a rectangle 500 in FIG. 5 ).
- the policy may be configured such that one or more areas within the environment may be avoided (e.g., obstacles 508 ) and one or more areas (e.g., targets 502 , 504 , 506 ) may be targeted.
- the targets may be associated with positive reinforcement and obstacles may be associated with negative reinforcement.
- a policy may comprise providing the adaptive apparatus (e.g., a robot) with an exemplary trajectory over which the robot is directed to follow as close as possible and/or points to which the robot is directed to reach during its path to the goal.
- the adaptive apparatus e.g., a robot
- training of adaptive robotic devices using demonstration and/or reinforcement learning may aid in placing the robot at an appropriate starting point and selecting an optimal trajectory.
- Clicker training described herein may be configured to enable an external agent (e.g., the teacher) to communicate an immediate cost C(x, t i ) at time t, to the robotic controller.
- the teacher may, based on observing operation of the adaptive apparatus, provide feedback via the reinforcement signal.
- the clicker 610 may comprise four buttons 612 .
- One pair of buttons may be used to provide positive reinforcement while another pair of buttons may be used to provide negative reinforcement.
- one button within the pair may be used to indicate hard constraints while the other button within the pair may be used to indicate soft constraints.
- the hard positive reinforcement may correspond to points along robot trajectory of operation that the robot must absolutely pass by or through (e.g., fuel station, docking station, and/or a target).
- the hard negative reinforcement may correspond to points along robot trajectory of operation that the robot absolutely must avoid (e.g., a cliff, an obstacle, a mine, hitting a part of robot body, and/or other points).
- the soft constraint may be used to indicate desirable areas for the robot to approach or areas that are not desirable.
- reaching the final goal may correspond to a hard positive reinforcement
- approaching the target along an acceptable trajectory may correspond to soft positive reinforcement
- going away from the target may correspond to a hard negative reinforcement
- deviation from the acceptable trajectory may correspond to a soft negative reinforcement.
- one button pair may be activated by the left hand while the other pair may be operated the right hand.
- the clicker realizations illustrated and described with respect to FIG. 6 are exemplary and various other clicker implementations comprising fewer (e.g., a single button) and/or more buttons may be utilized as well. Examples of other clicker implementations may include one or more of a touch phone interface of a tablet, smartphone, and/or other clicker implementations.
- the clicker may be embedded within the trainer apparatus.
- Such a “virtual” clicker may be configured to provide varying degrees of reinforcement gradations.
- the hard constraints may be used by the trainer to indicate initial and/or final points of a trajectory of a movement.
- the hard constraints may be used by the trainer to indicate cognitive sensorimotor states which are to be reached or avoided.
- the hard constraints may be used to indicate intermediate points that the adaptive controller is to pass by or through (e.g., a target destination, a detour point in order to go around an obstacle, and/or other intermediate points).
- the soft buttons may be used to indicate that the adaptive system is doing well or poorly, and/or whether the behavior that led to the reinforcement should be repeated or avoided in the future.
- Voice activated feedback may be utilized to provide reinforcement signal.
- words or short phrases may be used. Examples of such phrases may include one or more of “excellent”, “very good”, “good”, “bad”, “very bad”, “oh! never do that again”, and/or other phrases.
- the loudness and/or intonation of a voice command may be used to indicate the magnitude of the feedback.
- a continuous value may be used to describe the feedback.
- a continuous value may on a continuum from ⁇ 1 to +1, where ⁇ 1 is the largest negative reinforcement and +1 is the largest positive reinforcement.
- a joystick may be used to report a continuous value as feedback (e.g. most left at ⁇ 1 and most right at +1).
- the feedback may be given by moving arms, head, facial expressions, and/or other gestures. For example, “yes” (for positive reinforcement) may be indicated by moving the head up and down, while “no” (for negative reinforcement) may be indicated by moving the head left to right.
- the velocity of the movement of the gesture may indicate the magnitude of the desired feedback.
- the instantaneous cost C(q,t) may be configured equal 0 for the most positive feedback and +1 for the most negative feedback.
- the most negative feedback may be set to a large positive number, such as 10 8 .
- L denotes a decay parameter.
- the transformed ⁇ (q,t) may be subsequently used for constructing cost function F(q,t).
- FIGS. 3A-4 illustrate modification of instantaneous cost C(q,t) during training of the adaptive system.
- the curves shown in FIGS. 3A-4 may correspond to a trajectory of a robot (e.g., 512 , 514 of FIG. 5 ).
- FIG. 3A depicts C(q,t) as a function of the state parameter q corresponding to positive reinforcement that may be received when, e.g., the robot approaches a target area.
- the state parameter q may correspond to a spatial coordinate x and/or y and or other characteristic such as velocity, acceleration along these spatial coordinates, or visual, auditory, and/or motor signals.
- the curve 300 in FIG. 3A denotes a base level of cost in absence of reinforcement.
- the broken vertical lines (e.g., 312 ) in FIG. 3A denote instances when the reinforcement (e.g., click) is received.
- the curves 310 , 314 , 316 , 318 depict cost decreases due to receipt of positive reinforcement.
- Curves 316 , 318 illustrate use of exponential cost function adjustment as a function of state distance from the state at the clock event.
- the adjustment of curve 316 is characterized by a larger state space adjustment width 317 (e.g., 3 times in the example of FIG. 3A ) compared to the state space adjustment width 319 of the curve 318 .
- FIG. 3B depicts C(q,t) as a function of the state parameter q corresponding to negative reinforcement that may be received when, e.g., the robot approaches an obstacle and/or an area to be avoided.
- the broken vertical lines (e.g., 324 ) in FIG. 3A denote instances when the reinforcement (e.g., click) is received.
- the curves 320 , 326 , 330 , 334 depict cost increases due to receipt of negative reinforcement.
- Curves 330 , 336 illustrate use of exponential cost function increase as a function of state distance from the state associated with the clock event.
- the adjustment of the curve 330 is characterized by a larger state space adjustment width 332 (e.g., 3 times in the example of FIG. 3B ) compared to the state space adjustment width 336 of the curve 334 .
- FIG. 3C depicts C(q,t) as a function of the state parameter q corresponding to negative (curves 350 , 354 ) and positive reinforcement (curves 340 , 344 ) comprising Gaussian dependence of the cost adjustment on state distance from the state associated with the click event.
- the adjustment of curves 344 , 354 is characterized by a greater state space adjustment width 346 , 356 (e.g., 3 times in the example of FIG. 3C ) compared to the state space adjustment width 342 352 of curve 340 , 350 , respectively.
- cost modification may be configured using exponential distribution (e.g., curves 330 , 334 in FIG. 3B ). In some implementations that may require wider state extend and more gradual cost changes, cost modification may be configured using Gaussian distribution (e.g., curves 340 , 344 , 350 in FIG. 3C ).
- Individual modifications of instantaneous cost may be characterized by magnitude (e.g., 322 in FIG. 3B ) and spatial extent (e.g., 328 in FIG. 3B ).
- the magnitude may be configured based on the level of the reinforcement feedback (e.g., soft, hard, and/or other gradations as discussed above with respect to FIG. 6 ).
- the extent along the state coordinate q may correspond to, for example, spatial area of the obstacle, and/or be configured in accordance with traces of state coordinate activity, or the region in which previous changes occurred, or more generally, or regions that may be predictively associated with the clicked state coordinate and/or another parameter.
- Curves 400 410 in FIG. 4 illustrate time evolution of instantaneous cost C(q,t) along a trajectory (e.g., 512 , 514 in FIG. 5 ).
- the reinforcement input by the trainer e.g., via the clicker
- the adaptive system (e.g., 100 of FIG. 1 ) may be operated in accordance with a reinforcement algorithm (RL) process.
- the cost function F P (q,t) of the control policy may be constructed using one or more of the following constraints:
- the updated immediate cost may be back propagated to states experienced previously using a variety of techniques, such as dynamic programming, temporal-difference learning (TD), and/or accounting for in the cost function along a full trajectory, such as in the Path Integral for Policy Iteration (PI2)
- TD temporal-difference learning
- PI2 Path Integral for Policy Iteration
- the immediate cost may be affected in the state space vicinity of the clicker point (e.g., the extent 328 , 332 , 336 in FIG. 3B ).
- the extent may be localized (e.g., 328 ) and/or span a greater portion of the state space (e.g., global change) illustrated by the extent 356 of the curve 354 in FIG. 3C .
- the parameters A, B may be configured as a function of the magnitude of G(reinforce). For example, the parameters A, B may be configured such that A may increase as G increases and B may decrease as G increases.
- One or more curves (e.g., of Eqn. 7) may be applied to individual dimensions (e.g., degrees of freedom) of the state space.
- the instantaneous cost adaptation for the new cost may be set to zero at q click and to a large value for states away from q click .
- the large value may comprise the value of one.
- the large value may be selected equal 10 8 .
- the function may comprise a sharp convex function (e.g., a Gaussian).
- a sharp convex function e.g., a Gaussian
- Such a function may be an increasing function away from q click and may saturate after a distance ⁇ q from q click .
- the choice of function may be based on whether the positive points should have an effect on a large region of states or rather should be more local in state space.
- arm movements are configured more general and independent of final arm position, a local region may be selected.
- a larger region may be selected.
- the parameters A, B may be configured as a function of the magnitude of G(reinforce), for example such that A may increase as G increases and B may decrease as G increases.
- One or more curves (e.g., of Eqn. 7) may be applied to individual dimensions (e.g., degrees of freedom) of the state space.
- instantaneous cost adaptation the new cost may be set to a large value at q click and to zero for states away from q click .
- instantaneous cost adaptation may be performed based on general methodology of Eqn. 7-Eqn. 8 described above.
- the modulation parameters G, A, B for the soft constraint implementations may be selected to produce smaller changes in the instantaneous cost, compared to the hard constraint implementations.
- the concave/convex functions may also be configured not to cause abrupt changes in the (e.g., derivative dC(q,t)/dq ⁇ R_soft ⁇ R_hard, wherein R_soft/R_hard may be configured between 1/1000 and 1.
- the immediate cost C(q,t) configured based on, inter alia, reinforcement input from teacher, described above with respect to FIGS. 3A-4 , may be utilized to determine cost function (e.g., by the cost constructor 222 ) for use by a controller in order to cause robot movements to resemble movements of humans and/or animals.
- the immediate cost may contain other information that may be relevant to the task and target outcome (e.g., as minimization of energy, or passing around an obstacle).
- the cost-to-go for one movement i may be described as: C i *( , . . . ) ⁇ min T [C i ( T , . . . )+ ⁇ T], (Eqn. 9) where the duration of individual movement is given by:
- T i ( , ... ⁇ ) ⁇ ⁇ C i * ⁇ , ( Eqn . ⁇ 10 ) denotes the drive and is a constant of the movement.
- the drive value may be changed to change the character of the movement, for example from a slow, sluggish movement to an energetic movement.
- the drive although it can change, may be typically decided beforehand.
- An agent hand movement may be desired to be energetic, such as in karate agent moves, or perhaps slow for some movements of a ballerina agent.
- C i (T, . . . ) is the cost constraining the movement besides the drive, such that the hand moves through a particular point or towards a target, and how it may move to get there.
- the dependence on a range of arbitrary variables besides T is indicated by the three dots, “ . . . ”.
- a squared-jerk immediate cost may be utilized in order to determine the movement pattern.
- 2 2 , (Eqn. 13) where v and r are the hand velocity and position, respectively.
- C i
- 2 2 , (Eqn. 13) where v and r are the hand velocity and position, respectively.
- v and r are the hand velocity and position, respectively.
- the learning policy associated with optimizing the hand movement may be configured to minimize (i) the squared-jerk and (ii) the total time for the trajectory contemporaneously with one another.
- the drive conservation may be employed for time-invariant processes.
- the drive In the implementation of the squared-jerk, the drive may be expressed as:
- the desired movement may be determines using the following cost function: F* i ( , . . . ) ⁇ min T [F i ( T , . . . )+ ⁇ T]. (Eqn. 15)
- the constraint of reaching a target may be added with the final velocity of zero at the target in addition to minimizing the square-jerk in the drive equation above. Because it may be desired that the presence of the target does not influence the character of the rest of the movement, constraints may be configured to take effect in an area proximate the target so as to avoid cost modifications away from the target.
- the target area may be characterized in a state-space, where state may comprise, for example, coordinate, velocity, energy, and/or other parameters.
- I e - 0.5 * ( d d 0 ) 2 , ( Eqn . ⁇ 16 ) which may take on value of one near the target and value of zero away.
- O may be expressed as:
- an optimal trajectory may be based on a minimization of the amount of efforts (energy) used during the trajectory.
- An immediate cost related to the power, energy, and/or work spent during the movement may be added.
- GUI graphical user interface
- the GUI may be used to visualize and/or replay the clicker events along a trajectory (e.g., point 516 along trajectories 512 , 514 in FIG. 5 ) points along that were clicked.
- the GUI may visualize cost associated with individual events (e.g., positive/negative reinforcement) and/or one or more plant control parameters (e.g., speed, acceleration, energy use).
- the GUI may enable a user to remove and/or edit clicker events.
- the editing may comprise modifying one or more state parameters (e.g., x,y, coordinate, and/or acceleration in FIG. 5 ) of the controller.
- the editing may comprise increasing/decreasing the reward.
- Such transformations may be performed on a point by point basis, globally (all points), and/or based, for example, on an area of state space (e.g., all points within top left quadrant and/or with time between t1 and t2).
- Such editing methodology may enable the operator to evaluate individual training scenarios along the same trajectory without requiring repeat of training, thereby enabling to reduce training cost and/or duration.
- the operator may utilize the GUI in order to associate the clicker events with a particular state of the robot (e.g., instances of acceleration greater than 1 m/s 2 ).
- training may be based on a position of the robot, while in one or more implementations (e.g., obstacle approach/avoidance) the training may be may be based on other parameters (e.g., velocity and/or acceleration).
- the interface may enable to more specifically associate the clicker event with one or more control states (general state, sensorimotor state, and/or cognitive states), which were active some time before the clicker event.
- to be in a particular position away from the obstacle may not be is not as important (provided that the robot is not touching the obstacle) as the relative motion with respect to the object (e.g., velocity and/or acceleration).
- An operator may utilize the GUI to associate appropriate state parameters with the clicker event thereby accelerating learning.
- the graphical portion of the interface may be constructed in a manner that may be similar to existing movie editing tools (e.g., iMovie®) and/or robot control tools (e.g., Gostai Lab http://www.gostai.com/products/studio/gostai_lab), which may permit the visualization of camera and/or other signals from the robot on a computer screen
- movie editing tools e.g., iMovie®
- robot control tools e.g., Gostai Lab http://www.gostai.com/products/studio/gostai_lab
- the GUI may be configured to visualize individual and/or global cost functions, permit direct modifications of cost functions, via programming and/or graphical interface.
- the GUI may be utilized to add, remove, and/or replace trajectory starting point and/or target point (e.g., 504 , 506 in FIG. 5 ) thereby enabling the operator to evaluate individual training scenarios along different trajectories.
- trajectory starting point and/or target point e.g., 504 , 506 in FIG. 5
- FIGS. 8-9 illustrate methods for training of robotic devices, in accordance with one or more implementations.
- the operations of methods 800 , 900 presented below are intended to be illustrative. In some implementations, methods 800 , 900 may be accomplished with one or more additional operations not described, and/or without one or more of the operations discussed.
- the order in which the operations of methods 800 are illustrated in FIGS. 8-9 and described below is not intended to be limiting.
- Implementations of methods of FIGS. 8-9 may be realized in one or more processing devices (e.g., a digital processor, an analog processor, a digital circuit designed to process information, an analog circuit designed to process information, a state machine, and/or other mechanisms for electronically processing information).
- the one or more processing devices may include one or more devices executing some or all of the operations of methods 800 - 900 in response to instructions stored electronically on an electronic storage medium.
- the one or more processing devices may include one or more devices configured through hardware, firmware, and/or software to be specifically designed for execution of one or more of the operations of methods 800 - 900 .
- FIG. 8 one exemplary implementation of the reinforcement learning method of the disclosure for use with, for example, the robotic system 100 of FIG. 1 is described in detail.
- a determination may be made by the controller of the robotic system (e.g., controller 102 of system 100 in FIG. 1 ) as to whether reinforcement input is received.
- the reinforcement may positive or negative reinforcement, configured based on an evaluation of robotic system performance.
- the evaluation may be based on a squared jerk cost function methodology described above, configured to enable the robotic apparatus to move like an animal.
- the method may proceed to operation 804 where immediate cost may be adjusted in accordance with the reinforcement signal.
- the adjustment may comprise positive and/or negative components illustrated and described with respect to FIGS. 3A-4 , above.
- the cost function F(q,t) may be configured in accordance with the adjusted instantaneous cost C i (q,t). In one or more implementations, the cost function may be configured in accordance with Eqn. 12 described above.
- a controller may be configured.
- the controller may be configured in accordance with the updated cost function (e.g., Eqn. 12) and implements a control policy, which may comprise certain characteristics, such as minimizing energy use.
- the control policy and/or the controller may be determined using any of reinforcement learning (RL), cost optimization, dynamic programming, and/or other stochastic and/or deterministic optimization algorithms.
- RL reinforcement learning
- the RL algorithm may comprise a modified version of the path integral policy iteration algorithm with a cost function being constructed in part based on clicker training and in part based accordingly to requirements for the control policy (e.g., minimizing energy, minimizing jerk along trajectories, make the movement more animal/human-like).
- the updated cost function may modify the controller.
- the controller is an instantiation of the control policy, or the controller implement and applies the control policy, which is a more abstract concept.
- the control policy is a function which associates a motor output to states. It may be constructed by applying the controller to every state.
- the controller may be operated in accordance with the updated controller structure of step 808 .
- FIG. 9 illustrates an exemplary method of training a robotic device to move in a manner that is similar to movements of animal using reinforcement learning methodology of the disclosure.
- an evaluation of robotic system performance may be made.
- the evaluation may be based on a squared jerk cost function methodology described above, configured to enable the robotic apparatus to move like an animal.
- a determination may be made by the teacher as to whether operation of the robotic device is within target range. In some implementations, the determination may be based on whether the magnitude squared of the jerk is below maximum jerk level.
- the overall cost function may comprise a portion due to teacher intervention (e.g., clicker), and a portion configured based on a control policy.
- a control policy outcome may be quantifiable (e.g., minimum energy, and/or minimum jerk) the control policy cost portion may be computed automatically without teacher intervention.
- the control outcome may not be readily quantifiable (e.g., movement like a cat/human) clicker reinforcement may provide an online training method in order to achieve target behavior.
- the robotic device may learn sport moves (e.g., karate kicks) and/or dancing moves (e.g., ballet Arabesque).
- the teacher may grade the robot according to how close it performs to the target trajectory (e.g., karate, ballet).
- the method 900 may proceed to step 906 where a negative reinforcement may be provided to the robot.
- the target range may be configured in accordance with the task, the target control policy and configuration of a particular robot (e.g., arm length and number of joints) and may comprise a range for evaluation, which may be similar and/or as precise or imprecise as when interacting and teaching tasks to children and/or animals.
- the negative reinforcement may comprise hard and/or soft indication.
- the reinforcement signal may be provided using a clicker (e.g., 610 of FIG. 6 ) configured to deliver different degrees of negative indication (e.g., “bad”, “very bad”), as described in detail with respect to FIG. 6 above.
- the method 900 may proceed to step 909 where a positive reinforcement may be provided to the robot.
- the positive reinforcement may comprise hard and/or soft indication.
- the reinforcement signal may be provided using a clicker (e.g., 610 of FIG. 6 ) configured to deliver different degrees of reinforcement indication (e.g., “excellent”, “good”, “very good”), as described in detail with respect to FIG. 6 above.
- the cost function F(q,t) and/or the controller may be configured in accordance with the adjusted instantaneous cost C i (q,t).
- the cost function may be configured in accordance with Eqn. 12 described above
- the controller may be operated in accordance with the updated controller structure of step 910 .
- FIGS. 10-11C Various exemplary spiking network apparatus comprising one or more of the methods set forth herein (e.g., using the reinforcement learning mechanisms explained above) are now described with respect to FIGS. 10-11C .
- the computerized system 1000 of FIG. 10 may comprise an input interface 1010 , such as for example an image sensor, a computerized spiking retina, an audio array, a touch-sensitive input device, and/or other input interface.
- the input interface 1010 may be coupled to the processing block (e.g., a single or multi-processor block) via the input communication interface 1014 .
- the system 1000 may comprise a random access memory (RAM) 1008 , configured to store neuronal states and learning parameters (e.g., of Eqn. 1), and to facilitate learning.
- RAM random access memory
- the memory 1008 may be coupled to the processor 1002 via a direct connection (memory bus) 1016 .
- the memory 1008 may also be coupled to the processor 1002 via a high-speed processor bus 1012 .
- the system 1000 may comprise a nonvolatile storage device 1006 .
- the nonvolatile storage device 1006 may comprise, inter alia, computer readable instructions configured to implement various aspects of spiking neuronal network operation.
- the various aspects of spiking neuronal network operation may include one or more of sensory input encoding, connection plasticity, operation model of neurons, and/or other aspects of spiking neuronal network operation.
- the nonvolatile storage 1006 may be used to store state information of the neurons and connections when, for example, saving/loading network state snapshot, or implementing context switching.
- the context switching may comprise, for example, saving current network configuration for later use, and/or loading of a previously stored network configuration.
- the network configuration may comprise, inter alia, connection weights and update rules, neuronal states and learning rules, and/or other data.
- the computerized apparatus 1000 may be coupled to one or more external processing/storage/input devices via an I/O interface 1020 , such as a computer I/O bus (PCI-E), wired (e.g., Ethernet) or wireless (e.g., Wi-Fi) network connection.
- I/O interface 1020 such as a computer I/O bus (PCI-E), wired (e.g., Ethernet) or wireless (e.g., Wi-Fi) network connection.
- PCI-E computer I/O bus
- wired e.g., Ethernet
- wireless e.g., Wi-Fi
- the input/output interface may comprise a speech input (e.g., a microphone) and a speech recognition module configured to receive and recognize user commands.
- a speech input e.g., a microphone
- a speech recognition module configured to receive and recognize user commands.
- processing devices may be used with computerized system 1000 , including but not limited to, a single core/multicore CPU, DSP, FPGA, GPU, ASIC, combinations thereof, and/or other processors.
- Various user input/output interfaces may be similarly applicable to implementations of the invention including, for example, an LCD/LED monitor, touch-screen input and display device, speech input device, stylus, light pen, trackball, and/or other user I/O interfaces
- the neuromorphic processing system 1130 of FIG. 11A may comprise a plurality of processing blocks (micro-blocks) 1140 where individual micro cores may comprise a computing logic core 1132 and a memory block 1134 .
- the logic core 1132 may be configured to implement various aspects of neuronal node operation, such as the node model, and synaptic update and/or other tasks relevant to network operation.
- the memory block may be configured to store, inter alia, neuronal state variables and connection parameters (e.g., weights, delays, I/O mapping, and/or other connection parameters) of connections 1138 .
- the micro-blocks 1140 may be interconnected with one another using connections 1138 and routers 1136 .
- connections 1138 and routers 1136 may be interconnected with one another using connections 1138 and routers 1136 .
- connection layout in FIG. 11A is exemplary, and many other connection implementations (e.g., one to all, all-to-all, and/or other mapping) are compatible with the disclosure.
- the neuromorphic apparatus 1130 may be configured to receive input (e.g., sensory input) via the interface 1142 .
- input e.g., sensory input
- the apparatus 1130 may provide feedback information via the interface 1142 to facilitate encoding of the input signal.
- the interface 1142 may be utilized to receive reinforcement signal (e.g., 210 in FIG. 1 ).
- the neuromorphic apparatus 1130 may be configured to provide output via the interface 1144 .
- the output may comprise one or more control commands.
- the apparatus 1130 may interface to external fast response memory (e.g., RAM) via high bandwidth memory interface 1148 , thereby enabling storage of intermediate network operational parameters (e.g., spike timing, and/or other parameters.).
- external fast response memory e.g., RAM
- high bandwidth memory interface 1148 thereby enabling storage of intermediate network operational parameters (e.g., spike timing, and/or other parameters.).
- the apparatus 1130 may interface to external slower memory (e.g., Flash, or magnetic (hard drive)) via lower bandwidth memory interface 1146 , in order to facilitate program loading, operational mode changes, and retargeting, where network node and connection information for a current task is saved for future use and flushed, and previously stored network configuration is loaded in its place.
- FIG. 11B illustrates one or more implementations of shared bus neuromorphic computerized system comprising micro-blocks, described with respect to FIG. 11B , supra, architecture coupled to a shared 1140 .
- the apparatus 1145 of FIG. 11B may utilize one or more shared buses 1147 in order to interconnect micro-blocks 1140 with one another.
- FIG. 11C illustrates some implementations of cell-based neuromorphic computerized system architecture configured to implement efficient connection plasticity update mechanism in a spiking network is described in detail.
- the neuromorphic system 1150 of FIG. 11C may comprise a hierarchy of processing blocks (cells block).
- the lowest level L1 cell 1152 of the apparatus 1150 may comprise logic and memory and may be configured similar to the micro block 1140 of the apparatus shown in FIG. 11A .
- a number of cell blocks may be arranges in a cluster and communicate with one another a local interconnects 1162 , 1164 . Individual ones of such clusters may form higher level cell, e.g., cell L2, denoted as 1154 in FIG. 11 d .
- L2 clusters may communicate with one another via a second level interconnect 1166 and form a super-cluster L3, denoted as 1156 in FIG. 11D .
- the super-clusters 1154 may communicate via a third level interconnect 1168 and may form a next level cluster, and so on. It will be appreciated by those skilled in the arts that the hierarchical structure of the apparatus 1150 , comprising four cells-per-level, is merely one exemplary implementation, and other implementations may comprise more or fewer cells per level, and/or fewer or more levels.
- Different cell levels e.g., L1, L2, L3 of the apparatus 1150 may be configured to perform functionality various levels of complexity.
- different L1 cells may process in parallel different portions of the visual input (e.g., encode different frame macro-blocks), with the L2, L3 cells performing progressively higher level functionality (e.g., edge detection, object detection).
- Different L2, L3, cells may also perform different aspects of operating, for example, a robot, with one or more L2/L3 cells processing visual data from a camera, and other L2/L3 cells operating motor control block for implementing lens motion what tracking an object or performing lens stabilization functions.
- the neuromorphic apparatus 1150 may receive input (e.g., sensory input 228 of FIG. 2B ) via the interface 1160 .
- the interface 1160 may comprise a remote interface configured to receive reinforcement input (via e.g., a clicker 600 of FIG. 6 described above).
- the neuromorphic apparatus 1150 may provide output via the interface 1170 .
- the apparatus 1150 may perform all of the I/O functionality using single I/O block (not shown).
- the apparatus 1150 may interface to external fast response memory (e.g., RAM) via high bandwidth memory interface (not shown), thereby enabling storage of intermediate network operational parameters (e.g., spike timing, number of spikes, and/or other parameters.).
- external fast response memory e.g., RAM
- high bandwidth memory interface not shown
- intermediate network operational parameters e.g., spike timing, number of spikes, and/or other parameters.
- the apparatus 1150 may also interface to external slower memory (e.g., flash, or magnetic (hard drive)) via lower bandwidth memory interface (not shown), in order to facilitate program loading, operational mode changes, and retargeting, where network node and connection information for a current task may be saved for future use and flushed, and previously stored network configuration may be loaded in its place.
- the systems 1150 may combine (e.g., encode) inputs of different origin, type, dynamic range, noise signature, and/or other characteristics.
- input encoding may utilize analog-to-spiking signal conversion using a kernel expansion technique described in detail in co-owned U.S. patent application Ser. No. 13/623,842, entitled “SPIKING NEURON NETWORK ADAPTIVE CONTROL APPARATUS AND METHODS”, filed Sep. 20, 2012, incorporated herein by reference in its entirety.
- the terms “analog”, and/or “continuous” are used to describe a non-spiking signal, such as a continuous (e.g., analog) and/or discrete (e.g., digital signal).
- the continuous input signal may include one or more of an analog signal, a polyadic signal with arity greater than 2, an n-bit long discrete signal with n-bits greater than 2, a real-valued signal, and/or other signals.
- the advantage of the input encoding method described in the above cited application is to combine the advantage of sensory and state encoding in high dimension by a multitude of linear and nonlinear kernel functions and operators with a reinforcement learning controller, which can efficiently learn to make use of all the distributed and parallel input signals to construct a controller realizing an optimization of a performance function.
- the learning spiking network apparatus of the disclosure may be implemented as a software library configured to be executed by a computerized neural network apparatus (e.g., containing a digital processor).
- the learning apparatus may comprise a specialized hardware module (e.g., an embedded processor or controller).
- the spiking network apparatus may be implemented in a specialized or general purpose integrated circuit (e.g., ASIC, FPGA, PLD, and/or other integrated circuit). Myriad other implementations may exist that will be recognized by those of ordinary skill given the present disclosure.
- the reinforcement learning procedures according to the present disclosure can be used to simplify and improve control tasks for a wide assortment of control applications including, without limitation, industrial control, adaptive signal processing, navigation, and autonomous robotics.
- Autonomous robots may comprise robots that may perform tasks in unstructured environments without continuous human guidance.
- a high degree of robot autonomy may desirable in fields such as space exploration, cleaning floors, mowing lawns, and waste water treatment.
- Programming of autonomous robots may not be straightforward.
- Methodology described herein may be utilized in order to provide initial training of autonomous robots.
- the training may comprise an evaluation apparatus (e.g., a clicker).
- the clicker-based training methodology may enable users with little technical knowledge about design of a particular robot to train it.
- users having experience in training children and/or animals may employ similar approaches to training of robots utilizing clicker reinforcement learning process.
- Clicker training may be employed with robots comprising a plurality of changeable parts (e.g., chassis, sensors, limbs) so that a modified robot (e.g., after a limb replacement) may be re-trained to account for changes in the robot plant dynamics after part replacement.
- changeable parts e.g., chassis, sensors, limbs
- Clicker training may be employed without necessitating development of an accurate robot plant model while providing a mechanism for training through demonstration.
- clicker training may be configured to generate a cost function that may enable robot movements to resemble human motion. Such outcomes may permits easier acceptance of robots in human society, permits easier acceptance of robots by animals, easier acceptance, integration and aesthetics of robots for the service industry, including waiter robots, home servicing, home helping, and/or other applications. In some applications (to for example entertainment), training described herein may enable robots to move with more aesthetics, grace, human-like manner, enabling robots to be utilized in performances, dances, and/or other entertainment purposes.
- Exemplary implementations of the present disclosure may be useful in a variety of devices including without limitation prosthetic devices (such as artificial limbs), industrial control, autonomous and robotic apparatus, HVAC, and other electromechanical devices requiring accurate stabilization, set-point control, trajectory tracking functionality or other types of control.
- robotic devices may include manufacturing robots (e.g., automotive), military devices, and medical devices (e.g., for surgical robots).
- autonomous navigation may include rovers (e.g., for extraterrestrial, underwater, hazardous exploration environment), unmanned air vehicles, underwater vehicles, smart appliances (e.g., ROOMBA®), and/or robotic toys.
- adaptive signal processing systems comprising for example, artificial neural networks
- adaptive signal processing systems comprising for example, one or more or machine vision, pattern detection and pattern recognition, object classification, signal filtering, data segmentation, data compression, data mining, optimization and scheduling, complex mapping, and/or other adaptive signal processing systems.
Abstract
Description
where:
-
- θi(t) is an adaptation (learning) parameter of a synaptic connection between the pre-synaptic neuron i and the post-synaptic neuron j;
- η is a parameter referred to as the learning rate, η can be a constant parameter or it can be a function of some other system parameters;
- F(t) is the cost function (the cost-to-go function in some implementations); and
- ei(t) is eligibility trace, configured to characterize the relations between pre-synaptic and post-synaptic activity.
F=F(q(t),r(t)t),q(t)=q(x(t),y(t)), (Eqn. 2)
where:
-
- q(t) denotes controller state;
- x(t) denotes sensory input;
- y(t) denotes plant feedback; and
- r(t) denotes reinforcement input.
{circumflex over (C)}(q,t)=exp(−LC(q,t)). (Eqn. 3)
The instantaneous cost may be configured so that Ĉ(q,t)=1 for C(q,t)=0, 1 for Cost=0 and Ĉ(q,t)→0 as C(q,t)→∞1. In Eqn. 3, L denotes a decay parameter. The transformed Ĉ(q,t) may be subsequently used for constructing cost function F(q,t).
F(q,t)=F RL(q,t)+F P(q,t), (Eqn. 4)
where:
-
- FRL(q,t) denotes the cost function portion determined by the clicker training (described above); and
- FP(q,t) denotes the cost function portion determined according to requirements of the control policy.
The control policy may comprise one or more of minimizing energy, minimizing jerk along a trajectory, making movement of the robot to resemble animal- or human-like movements, and/or other criteria. As used herein the term “jerk” is used to refer to a temporal derivative of acceleration (e.g.,curve 720 shown inFIG. 7 ) associated with motion of the robot.
-
- constraint on jerk and/or higher order derivative terms;
- constraint on drive described below with respect to Eqn. 9; and/or
- constraint on energy.
-
- for a given number of degrees of freedom, individual state corresponding to the use of the clicker may be associated with a high cost (negative reinforcement) and/or small or zero cost (positive reinforcement)
-
- An increase in cost-to-go or immediate cost may comprise a “negative reward” and a decrease may comprise a “positive reward”;
-
- positive reinforcement may decrease the cost at the click point C(click+):
C(q click ,t+Δt)=C(q click ,t)−GC(q click ,t), (Eqn. 5) - negative reinforcement may increase the cost at the click point C(click−):
C(q click ,t+Δt)=C(q click ,t)+GC(q click ,t), (Eqn. 6)
where: - G is a parameter that may depend on the magnitude of the feedback provided, such that, e.g., G(very bad)>G(bad) and similarly, G(very good)>G(good);
- C(qclick,t+Δt) denotes the updated instantaneous cost; and
- C(qclick,t) denotes the initial instantaneous cost prior to reinforcement.
- positive reinforcement may decrease the cost at the click point C(click+):
C(q,t+Δt)=C(q,t)−A(q click −q)2 −B,
qε[q click −q1,q click +q2], and
q1++q2=Δq, (Eqn. 7)
where A and B are positive constants. In some implementations, such positive constants may be defined as A=104, or 108 and B=0. The parameters A, B may be configured as a function of the magnitude of G(reinforce). For example, the parameters A, B may be configured such that A may increase as G increases and B may decrease as G increases. One or more curves (e.g., of Eqn. 7) may be applied to individual dimensions (e.g., degrees of freedom) of the state space.
C(q,t+Δt)=C(q,t)+A(q click −q)2 +B,
qε[q click −q1,q click +q2], and
q1++q2=Δq, (Eqn. 8)
where A and B are positive constants. The parameter B may be used to provide a large cost (e.g., B=108) when q is at qclick. The parameters A, B may be configured as a function of the magnitude of G(reinforce), for example such that A may increase as G increases and B may decrease as G increases. One or more curves (e.g., of Eqn. 7) may be applied to individual dimensions (e.g., degrees of freedom) of the state space.
C i*(, . . . )≡minT [C i(T, . . . )+·T], (Eqn. 9)
where the duration of individual movement is given by:
denotes the drive and is a constant of the movement. The drive value may be changed to change the character of the movement, for example from a slow, sluggish movement to an energetic movement. The drive, although it can change, may be typically decided beforehand. An agent hand movement may be desired to be energetic, such as in karate agent moves, or perhaps slow for some movements of a ballerina agent. Ci(T, . . . ) is the cost constraining the movement besides the drive, such that the hand moves through a particular point or towards a target, and how it may move to get there. The dependence on a range of arbitrary variables besides T is indicated by the three dots, “ . . . ”.
F*(, . . . )=Σi=1 N F* i(, . . . ). (Eqn. 11)
Note that one form for the cost may be taken as
F i(T, . . . )=∫0 T C i(t, . . . )dt, (Eqn. 12)
where ci( . . . ) are the immediate costs.
C i =|{umlaut over (v)}| 2= 2, (Eqn. 13)
where v and r are the hand velocity and position, respectively. As shown in
and considered as a constant during the movements.
with these values, the drive may be set at =324. If a more energetic movement is desired, something ten times larger may be chosen, such as =3240, or any arbitrary value to obtain the desired movement character. With this new drive, the average speed v0 is about 0.3× 1/6.
F* i(, . . . )≡minT [F i(T, . . . )+·T]. (Eqn. 15)
which may take on value of one near the target and value of zero away. The term O may be expressed as:
which may take on value of zero near the target and value of one away, one may write the immediate costs in the following terms:
where the first term is the squared-jerk as above, the second term constraints a zero velocity at the target and the last term enforces the shortest path to the target, and d0 is a distance from the target, which defines the accuracy near the target. Kv and KP are constants weighting their relative constraints.
C i(t)=Σi=1 M K e V l(t)I l(t), (Eqn. 19)
where Vl(t) and Il(t) are respectively the voltage and current at the motor, M is the total number of motors, and Ke may be used to describe motor efficiency. In some implementations, the immediate cost may be expressed as:
C i(t)=Σi=1 M f(V l(t),I l(t)), (Eqn. 20)
where f( ) describes power characteristics of the motors.
W=∫F(l)dl,F=ma, (Eqn. 21)
where a denotes acceleration, and l denotes distance travel.
Claims (19)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/841,980 US8996177B2 (en) | 2013-03-15 | 2013-03-15 | Robotic training apparatus and methods |
PCT/US2014/026685 WO2014151926A2 (en) | 2013-03-15 | 2014-03-13 | Robotic training apparatus and methods |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/841,980 US8996177B2 (en) | 2013-03-15 | 2013-03-15 | Robotic training apparatus and methods |
Publications (2)
Publication Number | Publication Date |
---|---|
US20140277744A1 US20140277744A1 (en) | 2014-09-18 |
US8996177B2 true US8996177B2 (en) | 2015-03-31 |
Family
ID=51531446
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/841,980 Active 2033-12-01 US8996177B2 (en) | 2013-03-15 | 2013-03-15 | Robotic training apparatus and methods |
Country Status (2)
Country | Link |
---|---|
US (1) | US8996177B2 (en) |
WO (1) | WO2014151926A2 (en) |
Cited By (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140122398A1 (en) * | 2012-10-25 | 2014-05-01 | Brain Corporation | Modulated plasticity apparatus and methods for spiking neuron network |
US9186793B1 (en) | 2012-08-31 | 2015-11-17 | Brain Corporation | Apparatus and methods for controlling attention of a robot |
US9218563B2 (en) | 2012-10-25 | 2015-12-22 | Brain Corporation | Spiking neuron sensory processing apparatus and methods for saliency detection |
US9224090B2 (en) | 2012-05-07 | 2015-12-29 | Brain Corporation | Sensory input processing apparatus in a spiking neural network |
US9239985B2 (en) | 2013-06-19 | 2016-01-19 | Brain Corporation | Apparatus and methods for processing inputs in an artificial neuron network |
US9275326B2 (en) | 2012-11-30 | 2016-03-01 | Brain Corporation | Rate stabilization through plasticity in spiking neuron network |
US9311594B1 (en) | 2012-09-20 | 2016-04-12 | Brain Corporation | Spiking neuron network apparatus and methods for encoding of sensory data |
US9405975B2 (en) | 2010-03-26 | 2016-08-02 | Brain Corporation | Apparatus and methods for pulse-code invariant object recognition |
US9412041B1 (en) | 2012-06-29 | 2016-08-09 | Brain Corporation | Retinal apparatus and methods |
US9436909B2 (en) | 2013-06-19 | 2016-09-06 | Brain Corporation | Increased dynamic range artificial neuron network apparatus and methods |
US9481087B2 (en) * | 2014-12-26 | 2016-11-01 | National Chiao Tung University | Robot and control method thereof |
US9552546B1 (en) | 2013-07-30 | 2017-01-24 | Brain Corporation | Apparatus and methods for efficacy balancing in a spiking neuron network |
US9862092B2 (en) | 2014-03-13 | 2018-01-09 | Brain Corporation | Interface for use with trainable modular robotic apparatus |
US9873196B2 (en) | 2015-06-24 | 2018-01-23 | Brain Corporation | Bistatic object detection apparatus and methods |
US9881349B1 (en) | 2014-10-24 | 2018-01-30 | Gopro, Inc. | Apparatus and methods for computerized object identification |
US9987743B2 (en) | 2014-03-13 | 2018-06-05 | Brain Corporation | Trainable modular robotic apparatus and methods |
US10839302B2 (en) | 2015-11-24 | 2020-11-17 | The Research Foundation For The State University Of New York | Approximate value iteration with complex returns by bounding |
US10913154B2 (en) | 2018-01-02 | 2021-02-09 | General Electric Company | Systems and method for robotic learning of industrial tasks based on human demonstration |
US11442422B1 (en) * | 2016-03-03 | 2022-09-13 | AI Incorporated | Method for devising a schedule based on user input |
US11568236B2 (en) | 2018-01-25 | 2023-01-31 | The Research Foundation For The State University Of New York | Framework and methods of diverse exploration for fast and safe policy improvement |
US11631028B2 (en) | 2018-10-30 | 2023-04-18 | Samsung Electronics Co.. Ltd. | Method of updating policy for controlling action of robot and electronic device performing the method |
US11831955B2 (en) | 2010-07-12 | 2023-11-28 | Time Warner Cable Enterprises Llc | Apparatus and methods for content management and account linking across multiple content delivery networks |
Families Citing this family (36)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8775341B1 (en) | 2010-10-26 | 2014-07-08 | Michael Lamport Commons | Intelligent control with hierarchical stacked neural networks |
US9015093B1 (en) | 2010-10-26 | 2015-04-21 | Michael Lamport Commons | Intelligent control with hierarchical stacked neural networks |
US9566710B2 (en) * | 2011-06-02 | 2017-02-14 | Brain Corporation | Apparatus and methods for operating robotic devices using selective state space training |
JP6128767B2 (en) * | 2012-07-05 | 2017-05-17 | キヤノン株式会社 | Robot control apparatus and robot control method |
US9764468B2 (en) | 2013-03-15 | 2017-09-19 | Brain Corporation | Adaptive predictor apparatus and methods |
US9242372B2 (en) | 2013-05-31 | 2016-01-26 | Brain Corporation | Adaptive robotic interface apparatus and methods |
US9792546B2 (en) | 2013-06-14 | 2017-10-17 | Brain Corporation | Hierarchical robotic controller apparatus and methods |
US9314924B1 (en) | 2013-06-14 | 2016-04-19 | Brain Corporation | Predictive robotic controller apparatus and methods |
US9579789B2 (en) | 2013-09-27 | 2017-02-28 | Brain Corporation | Apparatus and methods for training of robotic control arbitration |
KR102000264B1 (en) * | 2013-10-01 | 2019-07-15 | 한국전자통신연구원 | Apparatus for inputting teaching data and apparatus and method for generating teaching command of robot |
US9463571B2 (en) | 2013-11-01 | 2016-10-11 | Brian Corporation | Apparatus and methods for online training of robots |
US9597797B2 (en) | 2013-11-01 | 2017-03-21 | Brain Corporation | Apparatus and methods for haptic training of robots |
US9248569B2 (en) | 2013-11-22 | 2016-02-02 | Brain Corporation | Discrepancy detection apparatus and methods for machine learning |
US9358685B2 (en) | 2014-02-03 | 2016-06-07 | Brain Corporation | Apparatus and methods for control of robot actions based on corrective user inputs |
CN104977927A (en) * | 2014-04-14 | 2015-10-14 | 科沃斯机器人科技(苏州)有限公司 | Surface treatment robot system |
US9346167B2 (en) | 2014-04-29 | 2016-05-24 | Brain Corporation | Trainable convolutional network apparatus and methods for operating a robotic vehicle |
US9630318B2 (en) | 2014-10-02 | 2017-04-25 | Brain Corporation | Feature detection apparatus and methods for training of robotic navigation |
US9717387B1 (en) | 2015-02-26 | 2017-08-01 | Brain Corporation | Apparatus and methods for programming and training of robotic household appliances |
US11100397B2 (en) | 2015-05-21 | 2021-08-24 | Rochester Institute Of Technology | Method and apparatus for training memristive learning systems |
TWI581731B (en) * | 2016-05-05 | 2017-05-11 | Solomon Tech Corp | Automatic shopping the method and equipment |
US10146334B2 (en) | 2016-06-09 | 2018-12-04 | Microsoft Technology Licensing, Llc | Passive optical and inertial tracking in slim form-factor |
US10146335B2 (en) | 2016-06-09 | 2018-12-04 | Microsoft Technology Licensing, Llc | Modular extension of inertial controller for six DOF mixed reality input |
US10807233B2 (en) | 2016-07-26 | 2020-10-20 | The University Of Connecticut | Skill transfer from a person to a robot |
JP6514166B2 (en) | 2016-09-16 | 2019-05-15 | ファナック株式会社 | Machine learning apparatus, robot system and machine learning method for learning robot operation program |
WO2019006586A1 (en) * | 2017-07-03 | 2019-01-10 | SZ DJI Technology Co., Ltd. | Neural network-based image target tracking by aerial vehicle |
US10646996B2 (en) * | 2017-07-21 | 2020-05-12 | Vicarious Fpc, Inc. | Methods for establishing and utilizing sensorimotor programs |
CN108406767A (en) * | 2018-02-13 | 2018-08-17 | 华南理工大学 | Robot autonomous learning method towards man-machine collaboration |
CN108563112A (en) * | 2018-03-30 | 2018-09-21 | 南京邮电大学 | Control method for emulating Soccer robot ball-handling |
JP7035734B2 (en) * | 2018-03-30 | 2022-03-15 | 富士通株式会社 | Reinforcement learning programs, reinforcement learning methods, and reinforcement learning devices |
CN108762249B (en) * | 2018-04-26 | 2019-11-08 | 常熟理工学院 | Clean robot optimum path planning method based on the optimization of approximate model multistep |
CN109324508B (en) * | 2018-09-04 | 2021-06-08 | 南宁学院 | Sewage treatment control method based on reinforcement learning |
CN109871011B (en) * | 2019-01-15 | 2022-03-11 | 哈尔滨工业大学(深圳) | Robot navigation method based on preprocessing layer and deep reinforcement learning |
CN109794937B (en) * | 2019-01-29 | 2021-10-01 | 南京邮电大学 | Football robot cooperation method based on reinforcement learning |
US11676064B2 (en) * | 2019-08-16 | 2023-06-13 | Mitsubishi Electric Research Laboratories, Inc. | Constraint adaptor for reinforcement learning control |
KR102090615B1 (en) * | 2019-11-19 | 2020-03-18 | 주식회사 파블로항공 | Drone Control System Using Model Predictive Control |
DE102020213527A1 (en) * | 2020-10-28 | 2022-04-28 | Robert Bosch Gesellschaft mit beschränkter Haftung | Method for optimizing a strategy for a robot |
Citations (69)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5063603A (en) | 1989-11-06 | 1991-11-05 | David Sarnoff Research Center, Inc. | Dynamic method for recognizing objects and image processing system therefor |
US5355435A (en) | 1992-05-18 | 1994-10-11 | New Mexico State University Technology Transfer Corp. | Asynchronous temporal neural processing element |
US5638359A (en) | 1992-12-14 | 1997-06-10 | Nokia Telecommunications Oy | Method for congestion management in a frame relay network and a node in a frame relay network |
US5673367A (en) | 1992-10-01 | 1997-09-30 | Buckley; Theresa M. | Method for neural network control of motion using real-time environmental feedback |
RU2108612C1 (en) | 1994-09-14 | 1998-04-10 | Круглов Сергей Петрович | Adaptive control system with identifier and implicit reference model |
US5875108A (en) | 1991-12-23 | 1999-02-23 | Hoffberg; Steven M. | Ergonomic man-machine interface incorporating adaptive pattern recognition based control system |
US6009418A (en) | 1996-05-02 | 1999-12-28 | Cooper; David L. | Method and apparatus for neural networking using semantic attractor architecture |
US6014653A (en) | 1996-01-26 | 2000-01-11 | Thaler; Stephen L. | Non-algorithmically implemented artificial neural networks and components thereof |
US6259988B1 (en) | 1998-07-20 | 2001-07-10 | Lockheed Martin Corporation | Real-time mission adaptable route planner |
US20010045809A1 (en) * | 2000-03-28 | 2001-11-29 | Yasushi Mukai | Teaching device for robots, robot using same, and robot operating method |
US20020038294A1 (en) | 2000-06-16 | 2002-03-28 | Masakazu Matsugu | Apparatus and method for detecting or recognizing pattern by employing a plurality of feature detecting elements |
US6458157B1 (en) | 1997-08-04 | 2002-10-01 | Suaning Gregg Joergen | Retinal stimulator |
US20030050903A1 (en) | 1997-06-11 | 2003-03-13 | Jim-Shih Liaw | Dynamic synapse for signal processing in neural networks |
US6545705B1 (en) | 1998-04-10 | 2003-04-08 | Lynx System Developers, Inc. | Camera with object recognition/data output |
US6546291B2 (en) | 2000-02-16 | 2003-04-08 | Massachusetts Eye & Ear Infirmary | Balance prosthesis |
US6545708B1 (en) | 1997-07-11 | 2003-04-08 | Sony Corporation | Camera controlling device and method for predicted viewing |
US6581046B1 (en) | 1997-10-10 | 2003-06-17 | Yeda Research And Development Co. Ltd. | Neuronal phase-locked loops |
US6760645B2 (en) * | 2001-04-30 | 2004-07-06 | Sony France S.A. | Training of autonomous robots |
US20040158358A1 (en) | 2003-02-06 | 2004-08-12 | Matsushita Electric Industrial Co., Ltd. | Method of teaching traveling path to robot and robot having function of learning traveling path |
US20040162638A1 (en) | 2002-08-21 | 2004-08-19 | Neal Solomon | System, method and apparatus for organizing groups of self-configurable mobile robotic agents in a multi-robotic system |
US20040193670A1 (en) | 2001-05-21 | 2004-09-30 | Langan John D. | Spatio-temporal filter and method |
US20050015351A1 (en) | 2003-07-18 | 2005-01-20 | Alex Nugent | Nanotechnology neural network methods and systems |
US20050036649A1 (en) | 2001-08-23 | 2005-02-17 | Jun Yokono | Robot apparatus, face recognition method, and face recognition apparatus |
US20050049749A1 (en) | 2003-08-27 | 2005-03-03 | Fanuc Ltd | Robot program position correcting apparatus |
US20050065651A1 (en) | 2003-07-24 | 2005-03-24 | Joseph Ayers | Process and architecture of robotic system to mimic animal behavior in the natural environment |
US20050283450A1 (en) | 2004-06-11 | 2005-12-22 | Masakazu Matsugu | Information processing apparatus, information processing method, pattern recognition apparatus, and pattern recognition method |
US20060069448A1 (en) * | 2002-08-30 | 2006-03-30 | Honda Giken Kogyo Kabushiki Kaisha | Joint device for artificial leg, method and control unit for controlling the joint device |
US20060161218A1 (en) | 2003-11-26 | 2006-07-20 | Wicab, Inc. | Systems and methods for treating traumatic brain injury |
US20070176643A1 (en) | 2005-06-17 | 2007-08-02 | Alex Nugent | Universal logic gate utilizing nanotechnology |
US20070208678A1 (en) | 2004-03-17 | 2007-09-06 | Canon Kabushiki Kaisha | Parallel Pulse Signal Processing Apparatus, Pattern Recognition Apparatus, And Image Input Apparatus |
JP4087423B2 (en) | 2006-10-17 | 2008-05-21 | 京セラミタ株式会社 | Portable communication device |
WO2008083335A2 (en) | 2006-12-29 | 2008-07-10 | Neurosciences Research Foundation, Inc. | Solving the distal reward problem through linkage of stdp and dopamine signaling |
US20080294074A1 (en) * | 2007-05-22 | 2008-11-27 | The Hong Kong Polytechnic University | Robotic training system with multi-orientation module |
US20090043722A1 (en) | 2003-03-27 | 2009-02-12 | Alex Nugent | Adaptive neural network utilizing nanotechnology-based components |
US20090287624A1 (en) | 2005-12-23 | 2009-11-19 | Societe De Commercialisation De Produits De La Recherche Applique-Socpra-Sciences Et Genie S.E.C. | Spatio-temporal pattern recognition using a spiking neural network and processing thereof on a portable and/or distributed computer |
US20100086171A1 (en) | 2008-10-02 | 2010-04-08 | Silverbrook Research Pty Ltd | Method of imaging coding pattern having merged data symbols |
US20100166320A1 (en) | 2008-12-26 | 2010-07-01 | Paquier Williams J F | Multi-stage image pattern recognizer |
US20100292835A1 (en) | 2009-05-15 | 2010-11-18 | Honda Research Institute Europe Gmbh | Autonomous robots with planning in unpredictable, dynamic and complex environments |
WO2010136961A1 (en) * | 2009-05-29 | 2010-12-02 | Koninklijke Philips Electronics N.V. | Control device and method for controlling a robot |
US7849030B2 (en) | 2006-05-31 | 2010-12-07 | Hartford Fire Insurance Company | Method and system for classifying documents |
US20110016071A1 (en) | 2009-07-20 | 2011-01-20 | Guillen Marcos E | Method for efficiently simulating the information processing in cells and tissues of the nervous system with a temporal series compressed encoding neural network |
US20110119214A1 (en) | 2009-11-18 | 2011-05-19 | International Business Machines Corporation | Area efficient neuromorphic circuits |
US20110119215A1 (en) | 2009-11-13 | 2011-05-19 | International Business Machines Corporation | Hardware analog-digital neural networks |
CN102226740A (en) | 2011-04-18 | 2011-10-26 | 中国计量学院 | Bearing fault detection method based on manner of controlling stochastic resonance by external periodic signal |
US20120011090A1 (en) | 2010-07-07 | 2012-01-12 | Qualcomm Incorporated | Methods and systems for three-memristor synapse with stdp and dopamine signaling |
US8145355B2 (en) * | 2001-08-31 | 2012-03-27 | Board Of Regents Of The Nevada System Of Higher Education, On Behalf Of The University Of Nevada, Reno | Coordinated joint motion control system |
US20120109866A1 (en) | 2010-10-29 | 2012-05-03 | International Business Machines Corporation | Compact cognitive synaptic computing circuits |
US20120143495A1 (en) | 2010-10-14 | 2012-06-07 | The University Of North Texas | Methods and systems for indoor navigation |
US20120173021A1 (en) * | 2009-09-28 | 2012-07-05 | Yuko Tsusaka | Control apparatus and control method for robot arm, robot, control program for robot arm, and robot arm control-purpose integrated electronic circuit |
US20120185092A1 (en) * | 2011-01-14 | 2012-07-19 | Hon Hai Precision Industry Co., Ltd. | Robotic arm position controlling device and robotic arm having same |
US20120209432A1 (en) | 2005-09-13 | 2012-08-16 | Neurosciences Research Foundation, Inc. | Hybrid control device |
US20120209428A1 (en) * | 2010-07-27 | 2012-08-16 | Kenji Mizutani | Motion path search device and method of searching for motion path |
US8315305B2 (en) | 2010-03-26 | 2012-11-20 | Brain Corporation | Systems and methods for invariant pulse latency coding |
US20120303091A1 (en) | 2010-03-26 | 2012-11-29 | Izhikevich Eugene M | Apparatus and methods for polychronous encoding and multiplexing in neuronal prosthetic devices |
US20120308136A1 (en) | 2010-03-26 | 2012-12-06 | Izhikevich Eugene M | Apparatus and methods for pulse-code invariant object recognition |
US20120308076A1 (en) | 2010-03-26 | 2012-12-06 | Filip Lukasz Piekniewski | Apparatus and methods for temporally proximate object recognition |
US20130073500A1 (en) | 2011-09-21 | 2013-03-21 | Botond Szatmary | High level neuromorphic network description apparatus and methods |
US20130073496A1 (en) | 2011-09-21 | 2013-03-21 | Botond Szatmary | Tag-based apparatus and methods for neural networks |
US20130073491A1 (en) | 2011-09-21 | 2013-03-21 | Eugene M. Izhikevich | Apparatus and methods for synaptic update in a pulse-coded network |
US20130116827A1 (en) | 2011-11-07 | 2013-05-09 | Seiko Epson Corporation | Robot control system, robot system, and sensor information processing apparatus |
US20130151450A1 (en) | 2011-12-07 | 2013-06-13 | Filip Ponulak | Neural network apparatus and methods for signal conversion |
US20130218821A1 (en) | 2011-09-21 | 2013-08-22 | Botond Szatmary | Round-trip engineering apparatus and methods for neural networks |
US20130325775A1 (en) | 2012-06-04 | 2013-12-05 | Brain Corporation | Dynamically reconfigurable stochastic learning apparatus and methods |
US20130325774A1 (en) | 2012-06-04 | 2013-12-05 | Brain Corporation | Learning stochastic apparatus and methods |
US20130325768A1 (en) | 2012-06-04 | 2013-12-05 | Brain Corporation | Stochastic spiking network learning apparatus and methods |
US20130325773A1 (en) | 2012-06-04 | 2013-12-05 | Brain Corporation | Stochastic apparatus and methods for implementing generalized learning rules |
US20140016858A1 (en) | 2012-07-12 | 2014-01-16 | Micah Richert | Spiking neuron network sensory processing apparatus and methods |
US20140089232A1 (en) | 2012-06-01 | 2014-03-27 | Brain Corporation | Neural network learning and collaboration apparatus and methods |
US8793205B1 (en) | 2012-09-20 | 2014-07-29 | Brain Corporation | Robotic learning and evolution apparatus |
-
2013
- 2013-03-15 US US13/841,980 patent/US8996177B2/en active Active
-
2014
- 2014-03-13 WO PCT/US2014/026685 patent/WO2014151926A2/en active Application Filing
Patent Citations (73)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5063603A (en) | 1989-11-06 | 1991-11-05 | David Sarnoff Research Center, Inc. | Dynamic method for recognizing objects and image processing system therefor |
US5875108A (en) | 1991-12-23 | 1999-02-23 | Hoffberg; Steven M. | Ergonomic man-machine interface incorporating adaptive pattern recognition based control system |
US5355435A (en) | 1992-05-18 | 1994-10-11 | New Mexico State University Technology Transfer Corp. | Asynchronous temporal neural processing element |
US5673367A (en) | 1992-10-01 | 1997-09-30 | Buckley; Theresa M. | Method for neural network control of motion using real-time environmental feedback |
US5638359A (en) | 1992-12-14 | 1997-06-10 | Nokia Telecommunications Oy | Method for congestion management in a frame relay network and a node in a frame relay network |
RU2108612C1 (en) | 1994-09-14 | 1998-04-10 | Круглов Сергей Петрович | Adaptive control system with identifier and implicit reference model |
US6014653A (en) | 1996-01-26 | 2000-01-11 | Thaler; Stephen L. | Non-algorithmically implemented artificial neural networks and components thereof |
US6009418A (en) | 1996-05-02 | 1999-12-28 | Cooper; David L. | Method and apparatus for neural networking using semantic attractor architecture |
US20030050903A1 (en) | 1997-06-11 | 2003-03-13 | Jim-Shih Liaw | Dynamic synapse for signal processing in neural networks |
US6545708B1 (en) | 1997-07-11 | 2003-04-08 | Sony Corporation | Camera controlling device and method for predicted viewing |
US6458157B1 (en) | 1997-08-04 | 2002-10-01 | Suaning Gregg Joergen | Retinal stimulator |
US6581046B1 (en) | 1997-10-10 | 2003-06-17 | Yeda Research And Development Co. Ltd. | Neuronal phase-locked loops |
US6545705B1 (en) | 1998-04-10 | 2003-04-08 | Lynx System Developers, Inc. | Camera with object recognition/data output |
US6259988B1 (en) | 1998-07-20 | 2001-07-10 | Lockheed Martin Corporation | Real-time mission adaptable route planner |
US6546291B2 (en) | 2000-02-16 | 2003-04-08 | Massachusetts Eye & Ear Infirmary | Balance prosthesis |
US20010045809A1 (en) * | 2000-03-28 | 2001-11-29 | Yasushi Mukai | Teaching device for robots, robot using same, and robot operating method |
US20020038294A1 (en) | 2000-06-16 | 2002-03-28 | Masakazu Matsugu | Apparatus and method for detecting or recognizing pattern by employing a plurality of feature detecting elements |
US6760645B2 (en) * | 2001-04-30 | 2004-07-06 | Sony France S.A. | Training of autonomous robots |
US20040193670A1 (en) | 2001-05-21 | 2004-09-30 | Langan John D. | Spatio-temporal filter and method |
US20050036649A1 (en) | 2001-08-23 | 2005-02-17 | Jun Yokono | Robot apparatus, face recognition method, and face recognition apparatus |
US8145355B2 (en) * | 2001-08-31 | 2012-03-27 | Board Of Regents Of The Nevada System Of Higher Education, On Behalf Of The University Of Nevada, Reno | Coordinated joint motion control system |
US20100286824A1 (en) | 2002-08-21 | 2010-11-11 | Neal Solomon | System for self-organizing mobile robotic collectives |
US20040162638A1 (en) | 2002-08-21 | 2004-08-19 | Neal Solomon | System, method and apparatus for organizing groups of self-configurable mobile robotic agents in a multi-robotic system |
US20060069448A1 (en) * | 2002-08-30 | 2006-03-30 | Honda Giken Kogyo Kabushiki Kaisha | Joint device for artificial leg, method and control unit for controlling the joint device |
US20040158358A1 (en) | 2003-02-06 | 2004-08-12 | Matsushita Electric Industrial Co., Ltd. | Method of teaching traveling path to robot and robot having function of learning traveling path |
US20090043722A1 (en) | 2003-03-27 | 2009-02-12 | Alex Nugent | Adaptive neural network utilizing nanotechnology-based components |
US20050015351A1 (en) | 2003-07-18 | 2005-01-20 | Alex Nugent | Nanotechnology neural network methods and systems |
US20050065651A1 (en) | 2003-07-24 | 2005-03-24 | Joseph Ayers | Process and architecture of robotic system to mimic animal behavior in the natural environment |
US20050049749A1 (en) | 2003-08-27 | 2005-03-03 | Fanuc Ltd | Robot program position correcting apparatus |
US20060161218A1 (en) | 2003-11-26 | 2006-07-20 | Wicab, Inc. | Systems and methods for treating traumatic brain injury |
US20070208678A1 (en) | 2004-03-17 | 2007-09-06 | Canon Kabushiki Kaisha | Parallel Pulse Signal Processing Apparatus, Pattern Recognition Apparatus, And Image Input Apparatus |
US20050283450A1 (en) | 2004-06-11 | 2005-12-22 | Masakazu Matsugu | Information processing apparatus, information processing method, pattern recognition apparatus, and pattern recognition method |
US8015130B2 (en) | 2004-06-11 | 2011-09-06 | Canon Kabushiki Kaisha | Information processing apparatus, information processing method, pattern recognition apparatus, and pattern recognition method |
US20070176643A1 (en) | 2005-06-17 | 2007-08-02 | Alex Nugent | Universal logic gate utilizing nanotechnology |
US20120209432A1 (en) | 2005-09-13 | 2012-08-16 | Neurosciences Research Foundation, Inc. | Hybrid control device |
US20090287624A1 (en) | 2005-12-23 | 2009-11-19 | Societe De Commercialisation De Produits De La Recherche Applique-Socpra-Sciences Et Genie S.E.C. | Spatio-temporal pattern recognition using a spiking neural network and processing thereof on a portable and/or distributed computer |
US7849030B2 (en) | 2006-05-31 | 2010-12-07 | Hartford Fire Insurance Company | Method and system for classifying documents |
JP4087423B2 (en) | 2006-10-17 | 2008-05-21 | 京セラミタ株式会社 | Portable communication device |
WO2008083335A2 (en) | 2006-12-29 | 2008-07-10 | Neurosciences Research Foundation, Inc. | Solving the distal reward problem through linkage of stdp and dopamine signaling |
US20080294074A1 (en) * | 2007-05-22 | 2008-11-27 | The Hong Kong Polytechnic University | Robotic training system with multi-orientation module |
US20100086171A1 (en) | 2008-10-02 | 2010-04-08 | Silverbrook Research Pty Ltd | Method of imaging coding pattern having merged data symbols |
US20100166320A1 (en) | 2008-12-26 | 2010-07-01 | Paquier Williams J F | Multi-stage image pattern recognizer |
US20100292835A1 (en) | 2009-05-15 | 2010-11-18 | Honda Research Institute Europe Gmbh | Autonomous robots with planning in unpredictable, dynamic and complex environments |
WO2010136961A1 (en) * | 2009-05-29 | 2010-12-02 | Koninklijke Philips Electronics N.V. | Control device and method for controlling a robot |
US20110016071A1 (en) | 2009-07-20 | 2011-01-20 | Guillen Marcos E | Method for efficiently simulating the information processing in cells and tissues of the nervous system with a temporal series compressed encoding neural network |
US20120173021A1 (en) * | 2009-09-28 | 2012-07-05 | Yuko Tsusaka | Control apparatus and control method for robot arm, robot, control program for robot arm, and robot arm control-purpose integrated electronic circuit |
US20110119215A1 (en) | 2009-11-13 | 2011-05-19 | International Business Machines Corporation | Hardware analog-digital neural networks |
US20110119214A1 (en) | 2009-11-18 | 2011-05-19 | International Business Machines Corporation | Area efficient neuromorphic circuits |
US20130251278A1 (en) | 2010-03-26 | 2013-09-26 | Eugene M. Izhikevich | Invariant pulse latency coding systems and methods |
US8315305B2 (en) | 2010-03-26 | 2012-11-20 | Brain Corporation | Systems and methods for invariant pulse latency coding |
US8467623B2 (en) | 2010-03-26 | 2013-06-18 | Brain Corporation | Invariant pulse latency coding systems and methods systems and methods |
US20120308076A1 (en) | 2010-03-26 | 2012-12-06 | Filip Lukasz Piekniewski | Apparatus and methods for temporally proximate object recognition |
US20120308136A1 (en) | 2010-03-26 | 2012-12-06 | Izhikevich Eugene M | Apparatus and methods for pulse-code invariant object recognition |
US20120303091A1 (en) | 2010-03-26 | 2012-11-29 | Izhikevich Eugene M | Apparatus and methods for polychronous encoding and multiplexing in neuronal prosthetic devices |
US20120011090A1 (en) | 2010-07-07 | 2012-01-12 | Qualcomm Incorporated | Methods and systems for three-memristor synapse with stdp and dopamine signaling |
US20120209428A1 (en) * | 2010-07-27 | 2012-08-16 | Kenji Mizutani | Motion path search device and method of searching for motion path |
US20120143495A1 (en) | 2010-10-14 | 2012-06-07 | The University Of North Texas | Methods and systems for indoor navigation |
US20120109866A1 (en) | 2010-10-29 | 2012-05-03 | International Business Machines Corporation | Compact cognitive synaptic computing circuits |
US20120185092A1 (en) * | 2011-01-14 | 2012-07-19 | Hon Hai Precision Industry Co., Ltd. | Robotic arm position controlling device and robotic arm having same |
CN102226740A (en) | 2011-04-18 | 2011-10-26 | 中国计量学院 | Bearing fault detection method based on manner of controlling stochastic resonance by external periodic signal |
US20130218821A1 (en) | 2011-09-21 | 2013-08-22 | Botond Szatmary | Round-trip engineering apparatus and methods for neural networks |
US20130073500A1 (en) | 2011-09-21 | 2013-03-21 | Botond Szatmary | High level neuromorphic network description apparatus and methods |
US20130073496A1 (en) | 2011-09-21 | 2013-03-21 | Botond Szatmary | Tag-based apparatus and methods for neural networks |
US20130073491A1 (en) | 2011-09-21 | 2013-03-21 | Eugene M. Izhikevich | Apparatus and methods for synaptic update in a pulse-coded network |
US20130116827A1 (en) | 2011-11-07 | 2013-05-09 | Seiko Epson Corporation | Robot control system, robot system, and sensor information processing apparatus |
US20130151450A1 (en) | 2011-12-07 | 2013-06-13 | Filip Ponulak | Neural network apparatus and methods for signal conversion |
US20140089232A1 (en) | 2012-06-01 | 2014-03-27 | Brain Corporation | Neural network learning and collaboration apparatus and methods |
US20130325775A1 (en) | 2012-06-04 | 2013-12-05 | Brain Corporation | Dynamically reconfigurable stochastic learning apparatus and methods |
US20130325774A1 (en) | 2012-06-04 | 2013-12-05 | Brain Corporation | Learning stochastic apparatus and methods |
US20130325768A1 (en) | 2012-06-04 | 2013-12-05 | Brain Corporation | Stochastic spiking network learning apparatus and methods |
US20130325773A1 (en) | 2012-06-04 | 2013-12-05 | Brain Corporation | Stochastic apparatus and methods for implementing generalized learning rules |
US20140016858A1 (en) | 2012-07-12 | 2014-01-16 | Micah Richert | Spiking neuron network sensory processing apparatus and methods |
US8793205B1 (en) | 2012-09-20 | 2014-07-29 | Brain Corporation | Robotic learning and evolution apparatus |
Non-Patent Citations (50)
Title |
---|
Asensio et al., "Robot Learning Control Based on Neural Network Prediction" ASME 8th Annual Dynamic Systems and Control Conference joint with the JSME 11th Motion and Vibration Conference 2012 [Retrieved on: Jun. 24, 2014]. Retrieved fro internet: . |
Asensio et al., "Robot Learning Control Based on Neural Network Prediction" ASME 8th Annual Dynamic Systems and Control Conference joint with the JSME 11th Motion and Vibration Conference 2012 [Retrieved on: Jun. 24, 2014]. Retrieved fro internet: <http://msc.berkeley.edu/wjchen/publications/DSC12—8726—FI.pdf>. |
Bohte, "Spiking Nueral Networks" Doctorate at the University of Leiden, Holland, Mar. 5, 2003, pp. 1-133 [retrieved on Nov. 14, 2012]. Retrieved from the internet: . |
Bohte, "Spiking Nueral Networks" Doctorate at the University of Leiden, Holland, Mar. 5, 2003, pp. 1-133 [retrieved on Nov. 14, 2012]. Retrieved from the internet: <URL: http://homepages.cwi.nl/-sbohte/publication/phdthesis.pdf>. |
Bouganis et al., Training a Spiking Neural Network to Control a 4-DoF Robotic Arm based on Spiking Timing-Dependent Plasticity in WCCI 2010 IEEE World Congress on Computational Intelligence Jul. 2010 [Retrieved on Jun. 24, 2014] Retrieved from internet: <http://www.doc.ic.ac.uk/-mpsha/IJCNN10a.pdf. |
Brette et al., Brian: a simple and flexible simulator for spiking neural networks, The Neuromorphic Engineer, Jul. 1, 2009, pp. 1-4, doi: 10.2417/1200906.1659. |
Cuntz et al., "One Rule to Grow Them All: A General Theory of Neuronal Branching and Its Paractical Application" PLOS Computational Biology, 6 (8), Published Aug. 5, 2010. |
Davison et al., PyNN: a common interface for neuronal network simulators, Frontiers in Neuroinformatics, Jan. 2009, pp. 1-10, vol. 2, Article 11. |
Djurfeldt, Mikael, The Connection-set Algebra: a formalism for the representation of connectivity structure in neuronal network models, implementations in Python and C++, and their use in simulators BMC Neuroscience Jul. 18, 2011 p. 1 12(Suppl 1):P80. |
Fidjeland et al., Accelerated Simulation of Spiking Neural Networks Using GPUs [online], 2010 [retrieved on Jun. 15, 2013], Retrieved from the Internet: URL:http://ieeexplore.ieee.org/xpls/abs-all.jsp?arnmber=5596678&tag=1. |
Fidjeland et al., Accelerated Simulation of Spiking Neural Networks Using GPUs [online], 2010 [retrieved on Jun. 15, 2013], Retrieved from the Internet: URL:http://ieeexplore.ieee.org/xpls/abs—all.jsp?arnmber=5596678&tag=1. |
Floreano et al., "Neuroevolution: from architectures to learning" Evol. Intel. Jan. 2008 1:47-62, [retrieved Dec. 30, 2013] [retrieved online from URL:. |
Floreano et al., "Neuroevolution: from architectures to learning" Evol. Intel. Jan. 2008 1:47-62, [retrieved Dec. 30, 2013] [retrieved online from URL:<http://inforscience.epfl.ch/record/112676/files/FloreanoDuerrMattiussi2008.pdf>. |
Gewaltig et al., NEST (Neural Simulation Tool), Scholarpedia, 2007, pp. 1-15, 2( 4 ): 1430, doi: 1 0.4249/scholarpedia.1430. |
Gleeson et al., ) NeuroML: A Language for Describing Data Driven Models of Neurons and Networks with a High Degree of Biological Detail, PLoS Computational Biology, Jun. 2010, pp. 1-19 vol. 6 Issue 6. |
Goodman et al., Brian: a simulator for spiking neural networks in Python, Frontiers in Neuroinformatics, Nov. 2008, pp. 1-10, vol. 2, Article 5. |
Gorchetchnikov et al., NineML: declarative, mathematically-explicit descriptions of spiking neuronal networks, Frontiers in Neuroinformatics, Conference Abstract: 4th INCF Congress of Neuroinformatics, doi: 1 0.3389/conf.fninf.2011.08.00098. |
Graham, Lyle J., The Surf-Hippo Reference Manual, http://www.neurophys.biomedicale.univparis5. fr/-graham/surf-hippo-files/Surf-Hippo%20Reference%20Manual.pdf, Mar. 2002, pp. 1-128. |
Huh et al., "Generalized Power Law for Curve Movements" 2011. |
Huh et al., "Real-Time Motor Control Using Recurrent Neural Networks" IEEEE Apr. 2009. |
Huh, "Rethinking Optimal Control of Human Movements" Thesis 2012. |
Izhikevich, "Polychronization: Computation with Spikes", Neural Computation, 25, 2006, 18, 245-282. |
Izhikevich, "Relating STDP to BCM", Neural Computation 15, 1511-1523 (2003). |
Izhikevich, "Simple Model of Spiking Neurons", IEEE Transactions on Neural Networks, vol. 14, No. 6, Nov. 2003, pp. 1569-1572. |
Karbowski et al., "Multispikes and Synchronization in a Large Neural Network with Temporal Delays", Neural Computation 12, 1573-1606 (2000). |
Kasabov, "Evolving Spiking Neural Networks for Spatio-and Spectro-Temporal Pattern Recognition", IEEE 6th International Conference Intelligent Systems 2012 [Retrieved on Jun. 24, 2014], Retrieved from internet: <http://ncs.ethz.ch/projects/evospike/publications/evolving-spiking-neural-networks-for-spatio-and-spectro-temporal-pattern-recognition-plenary-talk-ieee-is>. |
Khotanzad, "Classification of invariant image representations using a neural network" IEEF. Transactions on Acoustics, Speech, and Signal Processing, vol. 38, No. 6, Jun. 1990, pp. 1028-1038 [online], [retrieved on Dec. 10, 2013]. Retrieved from the Internet . |
Khotanzad, "Classification of invariant image representations using a neural network" IEEF. Transactions on Acoustics, Speech, and Signal Processing, vol. 38, No. 6, Jun. 1990, pp. 1028-1038 [online], [retrieved on Dec. 10, 2013]. Retrieved from the Internet <URL: http://www-ee.uta.edu/eeweb/IP/Courses/SPR/Reference/Khotanzad.pdf>. |
Laurent, "Issue 1-nnql-Refactor Nucleus into its own file-Neural Network Query Language" [retrieved on Nov. 12, 2013]. Retrieved from the Internet: URL:https://code.google.com/p/nnql/issues/detail?id=1. |
Laurent, "Issue 1—nnql—Refactor Nucleus into its own file—Neural Network Query Language" [retrieved on Nov. 12, 2013]. Retrieved from the Internet: URL:https://code.google.com/p/nnql/issues/detail?id=1. |
Laurent, "The Neural Network Query Language (NNQL) Reference" [retrieved on Nov. 12, 2013]. Retrieved from the Internet: URL:http://nnql.org/nnql.org. |
Mordatch et al., "Discovery of Complex Behaviors through Contract-Invariant Optimization" ACM Transactions on Graphics (TOG)-SIGGRAPH 2012 Conference. |
Mordatch et al., "Discovery of Complex Behaviors through Contract-Invariant Optimization" ACM Transactions on Graphics (TOG)—SIGGRAPH 2012 Conference. |
Nichols, A Reconfigurable Computing Architecture for Implementing Artificial Neural Networks on FPGA, Master's Thesis, The University of Guelph, 2003, pp. 1-235. |
Paugam-Moisy et al., "Computing with spiking neuron networks" G. Rozenberg T. Back, J. Kok (Eds.), Handbook of Natural Computing, Springer-Verlag (2010) [retrieved Dec. 30, 2013], [retrieved online from link.springer.com]. |
Pavlidis et al. Spiking neural network training using evolutionary algorithms. In: Proceedings 2005 IEEE International Joint Conference on Neural Networks, 2005. IJCNN'05, vol. 4, pp. 2190-2194 Publication Date Jul. 31, 2005 [online] [Retrieved on Dec. 10, 2013] Retrieved from the Internet <URL:http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.5.4346&rep=rep1&type=pdf. |
PCT International Search Report for PCT/US2014/026685 dated Oct. 3, 2014. |
PCT International Search Report for PCT/US2014/026738 dated Jul. 21, 2014. |
Pham et al., "Affine invariance of human hand movements: a direct test" 2012. |
Schaal et al., An Example Application of Policy Improvement with Path Integrals (PI2), Jun. 9, 2010. |
Schemmel et al., Implementing synaptic plasticity in a VLSI spiking neural network model in Proceedings of the 2006 International Joint Conference on Neural Networks (IJCNN'06), IEEE Press (2006) Jul. 16-21, 2006, pp. 1-6 [online], [retrieved on Dec. 10, 2013]. Retrieved from the Internet . |
Schemmel et al., Implementing synaptic plasticity in a VLSI spiking neural network model in Proceedings of the 2006 International Joint Conference on Neural Networks (IJCNN'06), IEEE Press (2006) Jul. 16-21, 2006, pp. 1-6 [online], [retrieved on Dec. 10, 2013]. Retrieved from the Internet <URL: http://www.kip.uni-heidelberg.de/veroeffentlichungen/download.cgi/4620/ps/1774.pdf>. |
Simulink® model [online], [Retrieved on Dec. 10, 2013] Retrieved from . |
Simulink® model [online], [Retrieved on Dec. 10, 2013] Retrieved from <URL: http://www.mathworks.com/products/simulink/index.html>. |
Sinyavskiy et al. "Reinforcement learning of a spiking neural network in the task of control of an agent in a virtual discrete environment" Rus. J. Nonlin. Dyn., 2011, vol. 7, No. 4 (Mobile Robots), pp. 859-875, chapters 1-8 (Russian Article with English Abstract). |
Sjostrom J., W. Gerstner. Spike-Timing Dependent Plasticity. Scholarpedia, [Online], 2010, 5(2), 1362. |
Szatmary et al., "Spike-timing Theory of Working Memory" PLoS Computational Biology, vol. 6, Issue 8, Aug. 19, 2010 [retrieved on Dec. 30, 2013]. Retrieved from the Internet: . |
Szatmary et al., "Spike-timing Theory of Working Memory" PLoS Computational Biology, vol. 6, Issue 8, Aug. 19, 2010 [retrieved on Dec. 30, 2013]. Retrieved from the Internet: <URL: http://www.ploscompbiol.org/article/info%3Adoi%2F10.1371%2Fjournal.pcbi.1000879#>. |
Tank, et al., "Neural computation by concentrating information in time", Proc. Nati. Acad. Sci. USA, vol. 84, Apr. 1987, pp. 1896-1900. |
Todorov "Direct Cortical Control of Muscle activation in voluntary arm movements: a model" Nature America Inc. http://neurosci.nature.com 2000. |
Cited By (32)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9405975B2 (en) | 2010-03-26 | 2016-08-02 | Brain Corporation | Apparatus and methods for pulse-code invariant object recognition |
US11831955B2 (en) | 2010-07-12 | 2023-11-28 | Time Warner Cable Enterprises Llc | Apparatus and methods for content management and account linking across multiple content delivery networks |
US9224090B2 (en) | 2012-05-07 | 2015-12-29 | Brain Corporation | Sensory input processing apparatus in a spiking neural network |
US9412041B1 (en) | 2012-06-29 | 2016-08-09 | Brain Corporation | Retinal apparatus and methods |
US10213921B2 (en) | 2012-08-31 | 2019-02-26 | Gopro, Inc. | Apparatus and methods for controlling attention of a robot |
US9186793B1 (en) | 2012-08-31 | 2015-11-17 | Brain Corporation | Apparatus and methods for controlling attention of a robot |
US10545074B2 (en) | 2012-08-31 | 2020-01-28 | Gopro, Inc. | Apparatus and methods for controlling attention of a robot |
US11867599B2 (en) | 2012-08-31 | 2024-01-09 | Gopro, Inc. | Apparatus and methods for controlling attention of a robot |
US11360003B2 (en) | 2012-08-31 | 2022-06-14 | Gopro, Inc. | Apparatus and methods for controlling attention of a robot |
US9311594B1 (en) | 2012-09-20 | 2016-04-12 | Brain Corporation | Spiking neuron network apparatus and methods for encoding of sensory data |
US9111226B2 (en) * | 2012-10-25 | 2015-08-18 | Brain Corporation | Modulated plasticity apparatus and methods for spiking neuron network |
US20140122398A1 (en) * | 2012-10-25 | 2014-05-01 | Brain Corporation | Modulated plasticity apparatus and methods for spiking neuron network |
US9218563B2 (en) | 2012-10-25 | 2015-12-22 | Brain Corporation | Spiking neuron sensory processing apparatus and methods for saliency detection |
US9275326B2 (en) | 2012-11-30 | 2016-03-01 | Brain Corporation | Rate stabilization through plasticity in spiking neuron network |
US9436909B2 (en) | 2013-06-19 | 2016-09-06 | Brain Corporation | Increased dynamic range artificial neuron network apparatus and methods |
US9239985B2 (en) | 2013-06-19 | 2016-01-19 | Brain Corporation | Apparatus and methods for processing inputs in an artificial neuron network |
US9552546B1 (en) | 2013-07-30 | 2017-01-24 | Brain Corporation | Apparatus and methods for efficacy balancing in a spiking neuron network |
US9987743B2 (en) | 2014-03-13 | 2018-06-05 | Brain Corporation | Trainable modular robotic apparatus and methods |
US10166675B2 (en) | 2014-03-13 | 2019-01-01 | Brain Corporation | Trainable modular robotic apparatus |
US10391628B2 (en) | 2014-03-13 | 2019-08-27 | Brain Corporation | Trainable modular robotic apparatus and methods |
US9862092B2 (en) | 2014-03-13 | 2018-01-09 | Brain Corporation | Interface for use with trainable modular robotic apparatus |
US11562458B2 (en) | 2014-10-24 | 2023-01-24 | Gopro, Inc. | Autonomous vehicle control method, system, and medium |
US10580102B1 (en) | 2014-10-24 | 2020-03-03 | Gopro, Inc. | Apparatus and methods for computerized object identification |
US9881349B1 (en) | 2014-10-24 | 2018-01-30 | Gopro, Inc. | Apparatus and methods for computerized object identification |
US9481087B2 (en) * | 2014-12-26 | 2016-11-01 | National Chiao Tung University | Robot and control method thereof |
US10807230B2 (en) | 2015-06-24 | 2020-10-20 | Brain Corporation | Bistatic object detection apparatus and methods |
US9873196B2 (en) | 2015-06-24 | 2018-01-23 | Brain Corporation | Bistatic object detection apparatus and methods |
US10839302B2 (en) | 2015-11-24 | 2020-11-17 | The Research Foundation For The State University Of New York | Approximate value iteration with complex returns by bounding |
US11442422B1 (en) * | 2016-03-03 | 2022-09-13 | AI Incorporated | Method for devising a schedule based on user input |
US10913154B2 (en) | 2018-01-02 | 2021-02-09 | General Electric Company | Systems and method for robotic learning of industrial tasks based on human demonstration |
US11568236B2 (en) | 2018-01-25 | 2023-01-31 | The Research Foundation For The State University Of New York | Framework and methods of diverse exploration for fast and safe policy improvement |
US11631028B2 (en) | 2018-10-30 | 2023-04-18 | Samsung Electronics Co.. Ltd. | Method of updating policy for controlling action of robot and electronic device performing the method |
Also Published As
Publication number | Publication date |
---|---|
WO2014151926A3 (en) | 2014-11-27 |
US20140277744A1 (en) | 2014-09-18 |
WO2014151926A2 (en) | 2014-09-25 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8996177B2 (en) | Robotic training apparatus and methods | |
US11161241B2 (en) | Apparatus and methods for online training of robots | |
US11331800B2 (en) | Adaptive predictor apparatus and methods | |
US9189730B1 (en) | Modulated stochasticity spiking neuron network controller apparatus and methods | |
US11279026B2 (en) | Reduced degree of freedom robotic controller apparatus and methods | |
US9367798B2 (en) | Spiking neuron network adaptive control apparatus and methods | |
US9082079B1 (en) | Proportional-integral-derivative controller effecting expansion kernels comprising a plurality of spiking neurons associated with a plurality of receptive fields | |
US20220212342A1 (en) | Predictive robotic controller apparatus and methods | |
US9597797B2 (en) | Apparatus and methods for haptic training of robots | |
US9566710B2 (en) | Apparatus and methods for operating robotic devices using selective state space training | |
US9384443B2 (en) | Robotic training apparatus and methods | |
US9008840B1 (en) | Apparatus and methods for reinforcement-guided supervised learning | |
US9579789B2 (en) | Apparatus and methods for training of robotic control arbitration | |
US20150005937A1 (en) | Action selection apparatus and methods | |
US20130325774A1 (en) | Learning stochastic apparatus and methods | |
US20140371912A1 (en) | Hierarchical robotic controller apparatus and methods | |
US11279025B2 (en) | Apparatus and methods for operating robotic devices using selective state space training | |
US20220203524A1 (en) | Apparatus and methods for operating robotic devices using selective state space training | |
WO2015116270A2 (en) | Reduced degree of freedom robotic controller apparatus and methods |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: BRAIN CORPORATION, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:COENEN, OLIVIER;REEL/FRAME:030110/0952 Effective date: 20130320 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
AS | Assignment |
Owner name: HERCULES CAPITAL, INC., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:BRAIN CORPORATION;REEL/FRAME:057851/0574 Effective date: 20211004 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
FEPP | Fee payment procedure |
Free format text: 7.5 YR SURCHARGE - LATE PMT W/IN 6 MO, LARGE ENTITY (ORIGINAL EVENT CODE: M1555); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |