Module: Curl¶
Simple support for many network protocols with libcurl
This module provides support for libcurl, enabling Chapel programs to work with many network protocols.
The curl homepage describes libcurl thus:
libcurl is a free and easy-to-use client-side URL transfer library, supporting
DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3,
POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet and TFTP. libcurl supports
SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload,
proxies, cookies, user+password authentication (Basic, Digest, NTLM,
Negotiate, Kerberos), file transfer resume, http proxy tunneling and more!
Dependencies¶
The Curl functionality in Chapel is dependent on libcurl. For information on how to install libcurl, see the curl installation instructions
The environment variables CHPL_AUXIO_INCLUDE and CHPL_AUXIO_LIBS must be set to point to the include and lib directories for libcurl respectively. More information on these variables can be found in auxIO.rst
Note
If libcurl is installed system-wide you should not need to set these variables.
Enabling Curl Support¶
Once you have ensured that libcurl is installed, and have the two variables above defined, set the environment variable CHPL_AUX_FILESYS to ‘curl’ to enable Curl support:
export CHPL_AUX_FILESYS=curl
Then, rebuild Chapel by executing ‘make’ from $CHPL_HOME:
make
Note
If Curl support is not enabled (which is the default), all features described below will compile successfully but will result in an error at runtime, saying: “No Curl Support”.
For information on how to enable and use Curl while also using other auxiliary IO extensions, as well as how to setup the CHPL_AUXIO_INCLUDE and CHPL_AUXIO_LIBS environment variables see doc/technotes/auxIO.rst in a Chapel release.
Using Curl Support in Chapel¶
- Chapel support for libcurl exposes two interfaces:
- A standard file I/O interface
- Bindings to libcurl’s curl_easy_setopt and curl_easy_perform
- Interface 1:
Since we do not support multiple readers on the same Curl handle, it is recommended that Chapel programs use openreader and openwriter in order to open up channels on a URL. Examples of how openreader and openwriter work can be found below. You may also open up a curl handle on a URL via open (url=..., mode=...), and then open up channels, or use the second interface described below to set options on that handle before opening up a channel on the handle. Note that in this last case, Chapel programs must not open multiple channels for the same file backed by a curl handle.
Note
The URL is the last parameter to open. Therefore when you are opening a URL if you are not using all of the parameters for open() or openreader/openwriter, you MUST specify all arguments by name, e.g.:
open(url="http://example.com", iomode.r);
will result in a compile time error. You would, instead, need to use:
open(url="http://example.com", mode=iomode.r).
- Interface 2:
Many times when we are connecting to a URL (FTP, IMAP, SMTP, HTTP) we have to give extra information to the Curl handle. This is done via the setopt() interface. Documentation on the various options, as well as the functions that are referenced below can be found here
Here are some simple code snippets demonstrating these two interfaces:
Example 1¶
// This example uses the first interface
var writer = openwriter("out.txt");
// Open a URL and get a reader on a section of the site
var reader = openreader(url="http://example.com");
var str:string;
// While we can read a line from example.com, write it to 'out.txt'
while(reader.readline(str)) do
writer.write(str);
Example 2¶
// This example uses the first interface
var str:string;
var reader = openreader(url="http://example.com");
reader.readstring(str);
reader.close();
// Write out to a URL via Curl
var writer = openwriter("http://127.0.0.1:1080");
writer.write(str);
writer.close();
Example 3¶
// Open a file on our local file system
var f = openwriter("out.txt");
// Now get a curl handle
var c = openreader(url="http://example.com");
var str:string;
// Read from example.com and write it out to out.txt
c.readstring(str);
f.write(str);
// Disconnect and free the curl handle and channel, and close the local file and
// channel
c.close();
f.close();
Example 4¶
// This example uses the second interface
// Open a file with a curl handle as the internal file
var c = open(url="http://example.com", mode=iomode.r);
// do a standard perform operation on the underlying curl handle
c.perform(); // This will print to stdout
// disconnect and free the curl handle
c.close();
Example 5¶
// This example uses the second interface + the first interface
var c = open(url="http://example.com", mode=iomode.r);
var str:string;
// Set verbose output from curl
c.setopt(curlopt_verbose, true);
// now read into the string
var reader = c.reader();
reader.readstring(str);
writeln(str);
reader.close();
c.close();
Example 6¶
// Connect to an IMAP site, and fetch mail from the inbox
config const username = "user";
config const password = "xxxx";
config const imapSite = "your_imap_site_here";
var handle = open(url=imapSite+"/INBOX/;UID=1", mode=iomode.r);
var reader = handle.reader();
var str:string;
handle.setopt((curlopt_username, username),
(curlopt_password, password));
//Calling this would give the same output
// handle.perform();
reader.readstring(str);
write(str);
reader.close();
handle.close();
Example 7¶
// This example shows a more complex example of how the two interfaces can work
// together: We use the second (setopt) interface to set the various options we
// need on a given curl handle, and then open a writer on that handle to write
// out via curl.
config const url = "smtp goes here";
config const from = "<some-email>";
config const to = "<some-email>";
config const cc = "<some-email>";
config const username = "username";
config const password = "password";
config const subject = "Testing Curl in Chapel (SMTP)";
config const message = "Hello! This is a message sent via Chapel!";
var handle = open(url=url, mode=iomode.cw);
var recipients:slist;
recipients.append(to);
recipients.append(cc);
var arr = "To: " + to + "\r\n" +
"From: " + from + "\r\n" +
"Cc: " + cc + "\r\n" +
"Subject: " + subject + "\r\n" + "\r\n" + message;
handle.setopt((curlopt_username , username),
(curlopt_password , password),
(curlopt_mail_from , from),
(curlopt_use_ssl , 3),
(curlopt_ssl_verifypeer , false),
(curlopt_ssl_verifyhost , false),
(curlopt_mail_rcpt , recipients.list),
(curlopt_verbose , true));
// Create a writer on the curl handle.
var writer = handle.writer();
// Now write out to Curl to send the email
writer.write(arr);
// recipients.free(); // BAD: This will free the data while it is in use!
writer.close();
// Note here how we free the slist AFTER we close the writer. This way we can
// ensure that we do not free data that is currently in use by curl.
recipients.free();
handle.close();
Curl Support Types and Functions¶
- proc file.setopt(opt: c_int, arg): bool¶
This function is the equivalent to the curl_easy_setopt function in libcurl. It sets information on the curl file handle that can change libcurl’s behavior.
Arguments: - opt – the curl option to set. This file makes available lower case versions of these curl options (e.g. use curlopt_url instead of CURLOPT_URL in your Chapel program)
- arg : int, string, bool, or slist – the value to set the curl option specified by opt.
- proc file.setopt(args ...?k)
Set curl options on a curl file. It is equivalent to the curl_setopt_array you might find in PHP.
For example, you might do:
curlfile.setopt((curlopt_username, username), (curlopt_password, password));
Arguments: args – any number of tuples of the form (curl_option, value). For each curl_option, this function will setopt it to its value.
- proc file.perform(): bool¶
Perform any blocking file transfer operations on the curl file. This function calls curl_easy_perform.
Corresponds to curl_easy_perform where the file has been opened up by specifying that url=<some url>.
Returns: true on success. This version halts if an error is encountered, but future versions will support returning an error code instead of halting.
- record slist¶
- A linked list of strings used in many curl setopt calls. This type corresponds to the libcurl type curl_slist.
Note
The slist type is not reference counted. Therefore the user is responsible for freeing the slist when they are done with it with slist.free.
- proc slist.append(str: string)¶
Append the string argument to an slist. This function is the same as calling curl_slist_append
This function halts if an error is encountered. Future versions will support returning an error code instead of halting.
Arguments: str – a string argument to append
- proc slist.free()¶
Free an slist. Chapel programs must call this function after using an slist. Programs must ensure that there are no ongoing connections using this slist when it is freed. For an example, see Example 7.
- const curlopt_file: c_int¶
- const curlopt_url: c_int¶
- const curlopt_port: c_int¶
- const curlopt_proxy: c_int¶
- const curlopt_userpwd: c_int¶
- const curlopt_proxyuserpwd: c_int¶
- const curlopt_range: c_int¶
- const curlopt_infile: c_int¶
- const curlopt_errorbuffer: c_int¶
- const curlopt_writefunction: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_WRITEFUNCTION.html
- const curlopt_readfunction: c_int¶
- const curlopt_timeout: c_int¶
- const curlopt_infilesize: c_int¶
- const curlopt_postfields: c_int¶
- const curlopt_referer: c_int¶
- const curlopt_ftpport: c_int¶
- const curlopt_useragent: c_int¶
- const curlopt_low_speed_limit: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_LOW_SPEED_LIMIT.html
- const curlopt_low_speed_time: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_LOW_SPEED_TIME.html
- const curlopt_resume_from: c_int¶
- const curlopt_httpheader: c_int¶
- const curlopt_httppost: c_int¶
- const curlopt_sslcert: c_int¶
- const curlopt_keypasswd: c_int¶
- const curlopt_crlf: c_int¶
- const curlopt_quote: c_int¶
- const curlopt_writeheader: c_int¶
- const curlopt_sslversion: c_int¶
- const curlopt_timecondition: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_TIMECONDITION.html
- const curlopt_timevalue: c_int¶
- const curlopt_customrequest: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_CUSTOMREQUEST.html
- const curlopt_stderr: c_int¶
- const curlopt_postquote: c_int¶
- const curlopt_writeinfo: c_int¶
This curlopt setting is deprecated
- const curlopt_verbose: c_int¶
- const curlopt_header: c_int¶
- const curlopt_noprogress: c_int¶
- const curlopt_nobody: c_int¶
- const curlopt_failonerror: c_int¶
- const curlopt_upload: c_int¶
- const curlopt_post: c_int¶
- const curlopt_dirlistonly: c_int¶
- const curlopt_append: c_int¶
- const curlopt_netrc: c_int¶
- const curlopt_followlocation: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_FOLLOWLOCATION.html
- const curlopt_transfertext: c_int¶
- const curlopt_put: c_int¶
- const curlopt_progressfunction: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_PROGRESSFUNCTION.html
- const curlopt_progressdata: c_int¶
- const curlopt_autoreferer: c_int¶
- const curlopt_proxyport: c_int¶
- const curlopt_postfieldsize: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_POSTFIELDSIZE.html
- const curlopt_httpproxytunnel: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_HTTPPROXYTUNNEL.html
- const curlopt_interface: c_int¶
- const curlopt_krblevel: c_int¶
- const curlopt_ssl_verifypeer: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html
- const curlopt_cainfo: c_int¶
- const curlopt_maxredirs: c_int¶
- const curlopt_filetime: c_int¶
- const curlopt_telnetoptions: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_TELNETOPTIONS.html
- const curlopt_maxconnects: c_int¶
- const curlopt_closepolicy: c_int¶
This option is deprecated
- const curlopt_fresh_connect: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_FRESH_CONNECT.html
- const curlopt_forbid_reuse: c_int¶
- const curlopt_random_file: c_int¶
- const curlopt_egdsocket: c_int¶
- const curlopt_connecttimeout: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_CONNECTTIMEOUT.html
- const curlopt_headerfunction: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_HEADERFUNCTION.html
- const curlopt_httpget: c_int¶
- const curlopt_ssl_verifyhost: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYHOST.html
- const curlopt_ssl_cipher_list: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_SSL_CIPHER_LIST.html
- const curlopt_http_version: c_int¶
- const curlopt_ftp_use_epsv: c_int¶
- const curlopt_sslcerttype: c_int¶
- const curlopt_sslkey: c_int¶
- const curlopt_sslkeytype: c_int¶
- const curlopt_sslengine: c_int¶
- const curlopt_sslengine_default: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_SSLENGINE_DEFAULT.html
- const curlopt_dns_use_global_cache: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_DNS_USE_GLOBAL_CACHE.html
- const curlopt_dns_cache_timeout: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_DNS_CACHE_TIMEOUT.html
- const curlopt_prequote: c_int¶
- const curlopt_debugfunction: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_DEBUGFUNCTION.html
- const curlopt_debugdata: c_int¶
- const curlopt_capath: c_int¶
- const curlopt_buffersize: c_int¶
- const curlopt_nosignal: c_int¶
- const curlopt_proxytype: c_int¶
- const curlopt_encoding: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html
- const curlopt_private: c_int¶
- const curlopt_http200aliases: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_HTTP200ALIASES.html
- const curlopt_unrestricted_auth: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_UNRESTRICTED_AUTH.html
- const curlopt_ftp_use_eprt: c_int¶
- const curlopt_httpauth: c_int¶
- const curlopt_ssl_ctx_function: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_SSL_CTX_FUNCTION.html
- const curlopt_ssl_ctx_data: c_int¶
- const curlopt_ftp_create_missing_dirs: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_FTP_CREATE_MISSING_DIRS.html
- const curlopt_proxyauth: c_int¶
- const curlopt_ftp_response_timeout: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_FTP_RESPONSE_TIMEOUT.html
- const curlopt_ipresolve: c_int¶
- const curlopt_maxfilesize: c_int¶
- const curlopt_infilesize_large: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_INFILESIZE_LARGE.html
- const curlopt_resume_from_large: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_RESUME_FROM_LARGE.html
- const curlopt_maxfilesize_large: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_MAXFILESIZE_LARGE.html
- const curlopt_netrc_file: c_int¶
- const curlopt_use_ssl: c_int¶
- const curlopt_postfieldsize_large: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_POSTFIELDSIZE_LARGE.html
- const curlopt_tcp_nodelay: c_int¶
- const curlopt_ftpsslauth: c_int¶
- const curlopt_ioctlfunction: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_IOCTLFUNCTION.html
- const curlopt_ioctldata: c_int¶
- const curlopt_ftp_account: c_int¶
- const curlopt_ignore_content_length: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_IGNORE_CONTENT_LENGTH.html
- const curlopt_ftp_skip_pasv_ip: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_FTP_SKIP_PASV_IP.html
- const curlopt_ftp_filemethod: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_FTP_FILEMETHOD.html
- const curlopt_localport: c_int¶
- const curlopt_localportrange: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_LOCALPORTRANGE.html
- const curlopt_connect_only: c_int¶
- const curlopt_conv_from_network_function: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_CONV_FROM_NETWORK_FUNCTION.html
- const curlopt_conv_to_network_function: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_CONV_TO_NETWORK_FUNCTION.html
- const curlopt_conv_from_utf8_function: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_CONV_FROM_UTF8_FUNCTION.html
- const curlopt_max_send_speed_large: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_MAX_SEND_SPEED_LARGE.html
- const curlopt_max_recv_speed_large: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_MAX_RECV_SPEED_LARGE.html
- const curlopt_ftp_alternative_to_user: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_FTP_ALTERNATIVE_TO_USER.html
- const curlopt_sockoptfunction: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_SOCKOPTFUNCTION.html
- const curlopt_sockoptdata: c_int¶
- const curlopt_ssl_sessionid_cache: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_SSL_SESSIONID_CACHE.html
- const curlopt_ssh_auth_types: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_SSH_AUTH_TYPES.html
- const curlopt_ssh_public_keyfile: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_SSH_PUBLIC_KEYFILE.html
- const curlopt_ssh_private_keyfile: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_SSH_PRIVATE_KEYFILE.html
- const curlopt_ftp_ssl_ccc: c_int¶
- const curlopt_timeout_ms: c_int¶
- const curlopt_connecttimeout_ms: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_CONNECTTIMEOUT_MS.html
- const curlopt_http_transfer_decoding: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_HTTP_TRANSFER_DECODING.html
- const curlopt_http_content_decoding: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_HTTP_CONTENT_DECODING.html
- const curlopt_new_file_perms: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_NEW_FILE_PERMS.html
- const curlopt_new_directory_perms: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_NEW_DIRECTORY_PERMS.html
- const curlopt_postredir: c_int¶
- const curlopt_ssh_host_public_key_md5: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_SSH_HOST_PUBLIC_KEY_MD5.html
- const curlopt_opensocketfunction: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_OPENSOCKETFUNCTION.html
- const curlopt_opensocketdata: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_OPENSOCKETDATA.html
- const curlopt_copypostfields: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_COPYPOSTFIELDS.html
- const curlopt_proxy_transfer_mode: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_PROXY_TRANSFER_MODE.html
- const curlopt_seekfunction: c_int¶
- const curlopt_seekdata: c_int¶
- const curlopt_crlfile: c_int¶
- const curlopt_issuercert: c_int¶
- const curlopt_address_scope: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_ADDRESS_SCOPE.html
- const curlopt_certinfo: c_int¶
- const curlopt_username: c_int¶
- const curlopt_password: c_int¶
- const curlopt_proxyusername: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_PROXYUSERNAME.html
- const curlopt_proxypassword: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_PROXYPASSWORD.html
- const curlopt_noproxy: c_int¶
- const curlopt_tftp_blksize: c_int¶
- const curlopt_socks5_gssapi_service: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_SOCKS5_GSSAPI_SERVICE.html
- const curlopt_socks5_gssapi_nec: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_SOCKS5_GSSAPI_NEC.html
- const curlopt_protocols: c_int¶
- const curlopt_redir_protocols: c_int¶
See http://curl.haxx.se/libcurl/c/CURLOPT_REDIR_PROTOCOLS.html
- const curlopt_lastentry: c_int¶
All other curlopt values will be less than this one
- const curlopt_mail_from: c_int¶
- const curlopt_mail_rcpt: c_int¶
- const curlopt_mail_auth: c_int¶