 |
 |
 |
HARK Cookbook |
 |
 |
|
|
|
|
HARK Cookbook |
|
|
|
Problem
I understand that sound source localization and sound source separation can be performed in HARK. How can I transfer the information to other systems? To answer this question, we consider the followings
Connection between two different HARK systems
Connection with other systems through a socket connection
Connection with ROS(Robot Operating System http://www.ros.org/wiki/)
Solution 1
Connection between two different HARK systems
Users may wish to process several HARK network files as submodules of one network file. For example, when performing parallel computations for the same algorithms, it is time consuming to describe all parallel module groups to one subnet sheet, and it would be fairly complicated since a large number of nodes must be treated. This section describes a method to treat one system described in the subnet sheet as one node.
Create the network to be connected
You may add a subnet sheet to the existing network file as
"Networks" -> "Add Network"
, or you may use an existing network file. In this case, make sure to designate inputs and outputs for this sheet (it is not a MUST since inputs and outputs can be changed after connecting).
Export the created subnet sheet (Note 1)
Make the created sheet active, and click
"Networks" -> "Export Network".
Give it an appropriate name and save the created subnet sheet as a network file. Since the subnet sheet name will be maintained when it is imported later, we recommend giving it a proper subnet sheet name before exporting.
Import the exported subnet sheet in the file to be connected
In the new file, the created subnet sheet becomes a single node. Click
"Networks" -> "Import Network"
and select the exported file. The file is read as a new subnet sheet.
Use it as a submodule
Open the sheet that differ from the imported sheet. Right click on the new sheet and select the same name as that of the created subnet sheet from
"New Node" -> "Subnet".
The module represents the subnet sheet itself with the same number of inputs and outputs as those of the imported subnet sheet.
Change of submodule (option)
The imported subnet sheet can be changed as wished. See Note 2 for examples.
Sub-modularization of this large system saves time into creating the same block construct and makes it easy to see the large-scale network file. The following are usage examples of this function.
Performing the same processing multiple times
When repeating the same processing or performing parallel computations, modular processing may allow the user to use the same node repeatedly in the main sheet.
Always using the processing as a template
If, for example, the same block configuration is always used for sound source localization, it would be easier to export the sound source localization processing once as a template and read it as one node every time, rather than to explicitly describe the processing of sound source localization every time.
(Note 1) Even without the exporting and importing functions, it is okay to copy and paste all nodes of a subnet sheet. Since, however, it is troublesome to copy and paste all the nodes of a large subnet sheet, it is recommended that the nodes be exported.
(Note 2) Modifications of submodules.
Changing the numbers of inputs and outputs and of names
The number of inputs/outputs and the names can be changed after importing. These changes require that you also change the main sheet to connect all inputs/outputs.
Changing parameters in a submodule one by one
Often, when using an imported submodule multiple times, the user will want to provide an argument, allowing the parameter set in the subnet sheet to be changed each time. If so, use "subnet_param", a property parameter type. Its usage is described in the training section material.
Solution 2
Connection with other systems through a socket connection
This chapter describes how to connect HARK systems with other systems by socket communication . The current HARK can be connected to Julius, a speech recognition engine, by socket communication. As a node for communication, the followings modules are in HARK by default.
Using these modules, HARK clients can send feature vectors to Julius as messages through a socket, allowing Julius to process these messages. For details, see the above two sources. However, since these source codes are somewhat complicated, the following section describes simple methods of making a HARK node, both as a client and as a server.
Solution 2-1 Connection with other systems through a socket connection (client)
In communicating with Julius, Julius continuously listens to messages from HARK as a server program. This example can be built by creating a simple server program and a HARK node. First create the server program listener.c, corresponding to a system other than HARK on the Julius side.
#include <stdio.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
main(int argc, char *argv[]){
int i;
int fd1, fd2;
struct sockaddr_in saddr;
struct sockaddr_in caddr;
int len;
int ret;
char buf[1024];
if (argc != 2){
printf("Usagelistener PORT_NUMBER\n");
exit(1);
}
if ((fd1 = socket(AF_INET, SOCK_STREAM, 0)) < 0){
perror("socket");
exit(1);
}
bzero((char *)&saddr, sizeof(saddr));
saddr.sin_family = AF_INET;
saddr.sin_addr.s_addr = INADDR_ANY;
saddr.sin_port = htons(atoi(argv[1]));
if (bind(fd1, (struct sockaddr *)&saddr, sizeof(saddr)) < 0){
perror("bind");
exit(1);
}
if (listen(fd1, 1)< 0){
perror("listen");
exit(1);
}
len = sizeof(caddr);
if ((fd2 = accept(fd1, (struct sockaddr *)&caddr, &len))< 0){
perror("accept");
exit(1);
}
close(fd1);
ret = read(fd2, buf, 1024);
while (strcmp(buf, "quit\n")!= 0){
printf("Received Message %s", buf);
ret = read(fd2, buf, 1024);
write(fd2, buf, 1024);
// This returns a responce.
}
close(fd2);
}
The content is a simple program for normal socket communication.
This program displays a message written in fd2. After cutting and pasting a source, compile it.
$ gcc -o listener listener.c
Next, create a client node of HARK. Cut and paste the following source to create TalkerTutorial.cc.
#include <iostream>
#include <BufferedNode.h>
#include <Buffer.h>
#include <string.h>
#include <sstream>
#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>
#include <csignal>
using namespace std;
using namespace FD;
class TalkerTutorial;
DECLARE_NODE(TalkerTutorial);
/*Node
*
* @name TalkerTutorial
* @category HARKDTutorial
* @description This block outputs the same integer as PARAM1 and sends it through socket.
*
* @output_name OUTPUT1
* @output_type int
* @output_description This output the same integer as PARAM1.
*
* @parameter_name PARAM1
* @parameter_type int
* @parameter_value 123
* @parameter_description Setting for OUTPUT1
*
* @parameter_name PORT
* @parameter_type int
* @parameter_value 8765
* @parameter_description Port number for socket connection
*
* @parameter_name IP_ADDR
* @parameter_type string
* @parameter_value 127.0.0.1
* @parameter_description IP address for socket connection
*
END*/
bool exit_flag2 = false;
class TalkerTutorial public BufferedNode {
int output1ID;
int param1;
int port;
string ip_addr;
struct sockaddr_in addr;
struct hostent *hp;
int fd;
int ret;
public
TalkerTutorial(string nodeName, ParameterSet params)
BufferedNode(nodeName, params){
output1ID= addOutput("OUTPUT1");
param1 =dereference_cast<int>(parameters.get("PARAM1"));
port= dereference_cast<int>(parameters.get("PORT"));
ip_addr = object_cast<String>(parameters.get("IP_ADDR"));
signal(SIGINT, signal_handler);
signal(SIGHUP, signal_handler);
signal(SIGPIPE, signal_handler);
if ((fd = socket(AF_INET, SOCK_STREAM, 0)) < 0){
perror("socket");
exit(1);
}
bzero((char *)&addr, sizeof(addr));
if ((hp = gethostbyname(ip_addr.c_str()))== NULL){
perror("No such host");
exit(1);
}
bcopy(hp->h_addr, &addr.sin_addr, hp->h_length);
addr.sin_family = AF_INET;
addr.sin_port = htons(port);
if (connect(fd, (struct sockaddr *)&addr, sizeof(addr)) < 0){
perror("connect");
exit(1);
}
inOrder = true;
}
void calculate(int output_id, int count, Buffer &out){
// Main loop routine starts here.
ostringstream message;
message << "[" << count << " , " << param1 << "]" << endl;
string buf = message.str();
write(fd, buf.c_str(), 1024);
cout << "Sent Message [" << count << " , " << param1 << "]" << endl;
(*(outputs[output1ID].buffer))[count] = ObjectRef(Int::alloc(param1));
if(exit_flag2){
cerr << "Operation closed..." << endl;
close(fd);
exit(1);
}
// Main loop routine ends here.
}
static void signal_handler(int s){
exit_flag2 = true;
}
};
The content of this node also consists of a client node for simple socket communication. Socket communication is performed for character strings stored in message for every count loop. After cutting and pasting, install it from the source compilation. When a server and a client are ready, build a network of Flowdesigner. This yields a simple node that performs socket communications for every specific time cycle in Sleep as shown below.
![\includegraphics[width=70mm]{fig/recipes/Advanced_TalkerPart.eps}](images/img-0082.png)
To execute, first start the server program. Decide an appropriate port number (8765 here) and execute with the following command line.
$ ./listener 8765
Next, launch the network file of Flowdesigner. Start a new console, and start the network file created in FlowDesigner above.
Left click the Sleep node > Set an appropriate cycle to SECONDS in float type (10000 here) Left click the TalkerTutorial node > Set an appropriate to PARAM1 in int type Left click the TalkerTutorial node > Set the port number to PORT in int type (8765 here) Left-click the TalkerTutorial node > Set an IP or a host name to IP_ADDR (127.0.0.1 here)
To set an IP address, we have assumed that both the server and the client work with the same machine. Of course, it is possible to communicate with a remote machine.
Click "Execute" button
Messages from HARK are then transferred to the operating server, where they are indicated in each console as.
Server side (listener)
Received Message [0 , 123] Received Message [1 , 123] Received Message [2 , 123] ...
Client side (Flowdesigner TalkerTutorial)
Sent Message [0 , 123] Sent Message [1 , 123] Sent Message [2 , 123] ...
Solution 2-1 Connection with other systems through a socket connection (server)
Next, create a server node that can receive data from a client program by socket communication. Similar to the solution above, a simple client program is prepared, followed by the creation of a server module. First create the server program Talker.c, a simple client program of socket communication.
#include <stdio.h>
#include <string.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>
main(int argc, char *argv[]){
struct sockaddr_in addr;
struct hostent *hp;
int fd;
int len;
int port;
char buf[1024];
int ret;
if (argc != 3){
printf("Usagetalker SERVER_NAME PORT_NUMBER\n");
exit(1);
}
if ((fd = socket(AF_INET, SOCK_STREAM, 0)) < 0){
perror("socket");
exit(1);
}
bzero((char *)&addr, sizeof(addr));
if ((hp = gethostbyname(argv[1])) == NULL){
perror("No such host");
exit(1);
}
bcopy(hp->h_addr, &addr.sin_addr, hp->h_length);
addr.sin_family = AF_INET;
addr.sin_port = htons(atoi(argv[2]));
if (connect(fd, (struct sockaddr *)&addr, sizeof(addr)) < 0){
perror("connect");
exit(1);
}
while (fgets(buf, 1024, stdin)){
write(fd, buf, 1024);
// ret = read(fd, buf, 1024); // This listens a responce.
// buf[ret] = '\0';
printf("Sent Message %s",buf);
}
close(fd);
exit(0);
}
Clearly, character strings read on a console are sent with the descriptor named fd. When the file is ready, compile it as
$ gcc -o talker talker.c
In building a server node in HARK, it is important to remember that the timing of messages sent by a client is unknown, making it necessary to create a server node so that a series of processes is performed every time a message is received. Therefore, it is necessary to create a in which the server itself becomes a trigger of new loop processing. For example
#include <iostream>
#include <BufferedNode.h>
#include <Buffer.h>
#include <string.h>
#include <sstream>
#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <csignal>
using namespace std;
using namespace FD;
class ListenerTutorial;
DECLARE_NODE(ListenerTutorial);
/*Node
*
* @name ListenerTutorial
* @category HARKDTutorial
* @description This block listens to messages from socket and outputs it.
*
* @output_name OUTPUT1
* @output_type string
* @output_description Same as the message received from socket.
*
* @output_name CONDITION
* @output_type bool
* @output_description True if we haven't reach the end of file yet.
*
* @parameter_name PORT
* @parameter_type int
* @parameter_value 8765
* @parameter_description Port number for socket connection
*
END*/
bool exit_flag = false;
class ListenerTutorial public BufferedNode {
int output1ID;
int conditionID;
int port;
int fd1, fd2;
struct sockaddr_in saddr;
struct sockaddr_in caddr;
int len;
int ret;
char buf[1024];
public
ListenerTutorial(string nodeName, ParameterSet params)
BufferedNode(nodeName, params){
output1ID= addOutput("OUTPUT1");
conditionID = addOutput("CONDITION");
port= dereference_cast<int>(parameters.get("PORT"));
signal(SIGINT, signal_handler);
signal(SIGHUP, signal_handler);
signal(SIGPIPE, signal_handler);
if ((fd1 = socket(AF_INET, SOCK_STREAM, 0)) < 0){
perror("socket");
exit(1);
}
bzero((char *)&saddr, sizeof(saddr));
saddr.sin_family = AF_INET;
saddr.sin_addr.s_addr = INADDR_ANY;
saddr.sin_port = htons(port);
if (bind(fd1, (struct sockaddr *)&saddr, sizeof(saddr)) < 0){
perror("bind");
exit(1);
}
if (listen(fd1, 1) < 0){
perror("listen");
exit(1);
}
len = sizeof(caddr);
if ((fd2 = accept(fd1, (struct sockaddr *)&caddr, (socklen_t *)&len)) < 0){
perror("accept");
exit(1);
}
close(fd1);
inOrder = true;
}
void calculate(int output_id, int count, Buffer &out){
// Main loop routine starts here.
Buffer &conditionBuffer = *(outputs[conditionID].buffer);
conditionBuffer[count]= (exit_flag ? FalseObject TrueObject);
ret = read(fd2, buf, 1024);
cout << "Count " << count << " , Received Message " << buf << endl;
ostringstream message;
message << buf << endl;
string output1 = message.str();
(*(outputs[output1ID].buffer))[count] = ObjectRef(new String(output1));
write(fd2, buf, 1024);
if(exit_flag){
cerr << "Operation closed..." << endl;
close(fd2);
exit(1);
}
// Main loop routine ends here.
}
static void signal_handler(int s){
exit_flag = true;
}
};
This node differs from a normal node, in that it adds a port that outputs a new bool variable named CONDITION. This CONDITION port always returns true except for forced terminations and pauses in socket communication. This port can trigger loops of the network file of Flowdesigner. After building a network file, and cutting and pasting the above source, it can be compiled and installed. Flowdesigner is then started, and the following node created
![\includegraphics[width=70mm]{fig/recipes/Advanced_ListenerPart.eps}](images/img-0083.png)
Here, the CONDITION output port became CONDITION of a loop of the network file. That is, the loop cycle of this network file was identical to the processing cycle of ListenerTutorial. The ListenerTutorial node suspends the processing until
ret = read(fd2, buf, 1024);
receives a new message. When a message is received, all calculate functions are processed and the processing of one count is completed. Making CONDITION of the network file the CONDITION port of the ListenerTutorial node enables the performance of a series of processes after one receipt of messages. To execute both, the server program (network file of Flowdesigner created above) is started.
Left-click the ListenerTutorial node > Set the port number for PORT (8765 here) Click the "Execute"
Make sure to set CONDITION. Start the client program next. Start the new console and move to a directory that stores talker.c, as compiled above. Execute it using the command line.
$ ./talker 127.0.0.1 8765
Assuming that the server and client work with the same machine, we have used the IP address 127.0.0.1, although it is possible to communicate with a remote machine. Now, enter some characters into the console of talker. Messages from the client console will be communicated to the server node of operating Flowdesigner. They are indicated in each console as (e.g. when inputting “hoge1”, “hoge2”, “hoge3”)
Server side (talker.c)
hoge1 Sent Message hoge1 hoge2 Sent Message hoge2 hoge3 Sent Message hoge3 ...
Client side (Flowdesigner ListenerTutorial))
Count 0 , Received Message hoge1 Count 1 , Received Message hoge2 Count 2 , Received Message hoge3 ...
This way, a HARK node itself can act as a server and receive a messages from other systems.
————
Solution 3
Connection with ROS (Robot Operating System)
If your system was created in ROS, the open source platform for robots developed by Willow Garage, you can create a communication between HARK and ROS using the instructions in this section.
Since a sufficient documents for ROS are available on the web, we assume here that (1) an ROS system has already been installed and, (2) the user knows the basic concepts of ROS, including publish to topic and subscribe from topic; i.e., we assume that the user has completed the Beginner Level of the ROS tutorial. Since there are two connection methods, each is described here. Moreover, Python is used for implementation of the ROS node.
(1) Use standard output This is a method that executes HARK as a subprocess in a node, utilizing the feature that a network of HARK can be solely executed. Based on this method, localization results from HARK are output as standard outputs (when wishing to have localization results, open the DEBUG property of LocalizeMUSIC ). For example, the network /home/hark/localize.n can be executed in python script with the following code.
import subprocess p = subprocess.Popen( "/home/hark/localize.n", cwd = "/home/hark/", stderr = subprocess.STDOUT, stdout = subprocess.PIPE)
Outputs of this network can be processed in python script with the code.
while not rospy.is_shutdown(): line = p.stdout.readline() Acquire information from the line
The information acquired from HARK can be published to an appropriate topic.
(2) Use of socket communication
Another method is to connect ROS and HARK via socket communication.
It will therefore be necessary to create a code of socket communication on the ROS side and a node on the HARK side. Despite difficulties, the entire configuration is clear for users. For creation of nodes for HARK, see the tutorial in the HARK documents. Some part of the python script, which receives information, is shown here.
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.bind( ('localhost', 12345)
) sock.listen(1) client, addr = sock.accept()
while 1
received = client.recv( 1024 )
(3) Communicate through topic
This is a communication method utilizing the ROS system. If a node to publish a message for a topic of ROS is created, localization results and separated sounds can be sent directly to ROS.
See Also
HARK document tutorial 3 (Creation of node)
|
|
|
HARK Cookbook |
|
|
|