Curl¶
Usage
use Curl;
Simple support for many network protocols with libcurl
This module provides support for libcurl, enabling Chapel programs to work with many network protocols.
The curl homepage describes libcurl thus:
libcurl is a free and easy-to-use client-side URL transfer library, supporting
DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3,
POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet and TFTP. libcurl supports
SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload,
proxies, cookies, user+password authentication (Basic, Digest, NTLM,
Negotiate, Kerberos), file transfer resume, http proxy tunneling and more!
Dependencies¶
- The Curl module in Chapel is dependent on libcurl. For more information
- on how to setup libcurl, see the Auxiliary I/O Systems page.
Using Curl Support in Chapel¶
- Chapel support for libcurl exposes two interfaces:
- A standard file I/O interface
- Bindings to libcurl's curl_easy_setopt and curl_easy_perform
- Interface 1:
Since we do not support multiple readers on the same Curl handle, it is recommended that Chapel programs use
openreader
andopenwriter
in order to open up channels on a URL. Examples of howopenreader
andopenwriter
work can be found below. You may also open up a curl handle on a URL viaopen
(url=..., mode=...), and then open up channels, or use the second interface described below to set options on that handle before opening up a channel on the handle. Note that in this last case, Chapel programs must not open multiple channels for the same file backed by a curl handle.Note
The URL is the last parameter to
open
. Therefore when you are opening a URL if you are not using all of the parameters foropen()
oropenreader
/openwriter
, you MUST specify all arguments by name, e.g.:open(url="https://example.com", iomode.r);
will result in a compile time error. You would, instead, need to use:
open(url="https://example.com", mode=iomode.r).
- Interface 2:
Many times when we are connecting to a URL (FTP, IMAP, SMTP, HTTP) we have to give extra information to the Curl handle. This is done via the setopt() interface. Documentation on the various options, as well as the functions that are referenced below can be found here
Here are some simple code snippets demonstrating these two interfaces:
Example 1¶
// This example uses the first interface
var writer = openwriter("out.txt");
// Open a URL and get a reader on a section of the site
var reader = openreader(url="https://example.com");
var str:string;
// While we can read a line from example.com, write it to 'out.txt'
while(reader.readline(str)) do
writer.write(str);
Example 2¶
// This example uses the first interface
var str:string;
var reader = openreader(url="https://example.com");
reader.readstring(str);
reader.close();
// Write out to a URL via Curl
var writer = openwriter("https://127.0.0.1:1080");
writer.write(str);
writer.close();
Example 3¶
// Open a file on our local file system
var f = openwriter("out.txt");
// Now get a curl handle
var c = openreader(url="https://example.com");
var str:string;
// Read from example.com and write it out to out.txt
c.readstring(str);
f.write(str);
// Disconnect and free the curl handle and channel, and close the local file and
// channel
c.close();
f.close();
Example 4¶
// This example uses the second interface
// Open a file with a curl handle as the internal file
var c = open(url="https://example.com", mode=iomode.r);
// do a standard perform operation on the underlying curl handle
c.perform(); // This will print to stdout
// disconnect and free the curl handle
c.close();
Example 5¶
// This example uses the second interface + the first interface
var c = open(url="https://example.com", mode=iomode.r);
var str:string;
// Set verbose output from curl
c.setopt(curlopt_verbose, true);
// now read into the string
var reader = c.reader();
reader.readstring(str);
writeln(str);
reader.close();
c.close();
Example 6¶
// Connect to an IMAP site, and fetch mail from the inbox
config const username = "user";
config const password = "xxxx";
config const imapSite = "your_imap_site_here";
var handle = open(url=imapSite+"/INBOX/;UID=1", mode=iomode.r);
var reader = handle.reader();
var str:string;
handle.setopt((curlopt_username, username),
(curlopt_password, password));
//Calling this would give the same output
// handle.perform();
reader.readstring(str);
write(str);
reader.close();
handle.close();
Example 7¶
// This example shows a more complex example of how the two interfaces can work
// together: We use the second (setopt) interface to set the various options we
// need on a given curl handle, and then open a writer on that handle to write
// out via curl.
config const url = "smtp goes here";
config const from = "<some-email>";
config const to = "<some-email>";
config const cc = "<some-email>";
config const username = "username";
config const password = "password";
config const subject = "Testing Curl in Chapel (SMTP)";
config const message = "Hello! This is a message sent via Chapel!";
var handle = open(url=url, mode=iomode.cw);
var recipients:slist;
recipients.append(to);
recipients.append(cc);
var arr = "To: " + to + "\r\n" +
"From: " + from + "\r\n" +
"Cc: " + cc + "\r\n" +
"Subject: " + subject + "\r\n" + "\r\n" + message;
handle.setopt((curlopt_username , username),
(curlopt_password , password),
(curlopt_mail_from , from),
(curlopt_use_ssl , 3),
(curlopt_ssl_verifypeer , false),
(curlopt_ssl_verifyhost , false),
(curlopt_mail_rcpt , recipients.list),
(curlopt_verbose , true));
// Create a writer on the curl handle.
var writer = handle.writer();
// Now write out to Curl to send the email
writer.write(arr);
// recipients.free(); // BAD: This will free the data while it is in use!
writer.close();
// Note here how we free the slist AFTER we close the writer. This way we can
// ensure that we do not free data that is currently in use by curl.
recipients.free();
handle.close();
Curl Support Types and Functions¶
-
proc
file.
setopt
(opt: c_int, arg): bool throws¶ This function is the equivalent to the curl_easy_setopt function in libcurl. It sets information on the curl file handle that can change libcurl's behavior.
Arguments: - opt -- the curl option to set. This file makes available lower case versions of these curl options (e.g. use curlopt_url instead of CURLOPT_URL in your Chapel program)
- arg : int, string, bool, or slist -- the value to set the curl option specified by opt.
-
proc
file.
setopt
(args ...?k) Set curl options on a curl file. It is equivalent to the curl_setopt_array you might find in PHP.
For example, you might do:
curlfile.setopt((curlopt_username, username), (curlopt_password, password));
Arguments: args -- any number of tuples of the form (curl_option, value). For each curl_option, this function will setopt it to its value.
-
proc
file.
perform
(): bool throws¶ Perform any blocking file transfer operations on the curl file. This function calls curl_easy_perform.
Corresponds to curl_easy_perform where the file has been opened up by specifying that url=<some url>.
Returns: true on success. This version halts if an error is encountered, but future versions will support returning an error code instead of halting.
-
record
slist
¶ - A linked list of strings used in many curl setopt calls. This type corresponds to the libcurl type curl_slist.
Note
The slist type is not reference counted. Therefore the user is responsible for freeing the slist when they are done with it with
slist.free
.
-
proc
slist.
append
(str: string) throws¶ Append the string argument to an slist. This function is the same as calling curl_slist_append
This function halts if an error is encountered. Future versions will support returning an error code instead of halting.
Arguments: str -- a string argument to append
-
proc
slist.
free
()¶ Free an slist. Chapel programs must call this function after using an slist. Programs must ensure that there are no ongoing connections using this slist when it is freed. For an example, see Example 7.
-
const
curlopt_file
: c_int¶
-
const
curlopt_url
: c_int¶
-
const
curlopt_port
: c_int¶
-
const
curlopt_proxy
: c_int¶
-
const
curlopt_userpwd
: c_int¶
-
const
curlopt_proxyuserpwd
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_PROXYUSERPWD.html
-
const
curlopt_range
: c_int¶
-
const
curlopt_infile
: c_int¶
-
const
curlopt_errorbuffer
: c_int¶
-
const
curlopt_writefunction
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_WRITEFUNCTION.html
-
const
curlopt_readfunction
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_READFUNCTION.html
-
const
curlopt_timeout
: c_int¶
-
const
curlopt_infilesize
: c_int¶
-
const
curlopt_postfields
: c_int¶
-
const
curlopt_referer
: c_int¶
-
const
curlopt_ftpport
: c_int¶
-
const
curlopt_useragent
: c_int¶
-
const
curlopt_low_speed_limit
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_LOW_SPEED_LIMIT.html
-
const
curlopt_low_speed_time
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_LOW_SPEED_TIME.html
-
const
curlopt_resume_from
: c_int¶
-
const
curlopt_httpheader
: c_int¶
-
const
curlopt_httppost
: c_int¶
-
const
curlopt_sslcert
: c_int¶
-
const
curlopt_keypasswd
: c_int¶
-
const
curlopt_crlf
: c_int¶
-
const
curlopt_quote
: c_int¶
-
const
curlopt_writeheader
: c_int¶
-
const
curlopt_sslversion
: c_int¶
-
const
curlopt_timecondition
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_TIMECONDITION.html
-
const
curlopt_timevalue
: c_int¶
-
const
curlopt_customrequest
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_CUSTOMREQUEST.html
-
const
curlopt_stderr
: c_int¶
-
const
curlopt_postquote
: c_int¶
-
const
curlopt_writeinfo
: c_int¶ This curlopt setting is deprecated
-
const
curlopt_verbose
: c_int¶
-
const
curlopt_header
: c_int¶
-
const
curlopt_noprogress
: c_int¶
-
const
curlopt_nobody
: c_int¶
-
const
curlopt_failonerror
: c_int¶
-
const
curlopt_upload
: c_int¶
-
const
curlopt_post
: c_int¶
-
const
curlopt_dirlistonly
: c_int¶
-
const
curlopt_append
: c_int¶
-
const
curlopt_netrc
: c_int¶
-
const
curlopt_followlocation
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_FOLLOWLOCATION.html
-
const
curlopt_transfertext
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_TRANSFERTEXT.html
-
const
curlopt_put
: c_int¶
-
const
curlopt_progressfunction
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_PROGRESSFUNCTION.html
-
const
curlopt_progressdata
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_PROGRESSDATA.html
-
const
curlopt_autoreferer
: c_int¶
-
const
curlopt_proxyport
: c_int¶
-
const
curlopt_postfieldsize
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_POSTFIELDSIZE.html
-
const
curlopt_httpproxytunnel
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_HTTPPROXYTUNNEL.html
-
const
curlopt_interface
: c_int¶
-
const
curlopt_krblevel
: c_int¶
-
const
curlopt_ssl_verifypeer
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html
-
const
curlopt_cainfo
: c_int¶
-
const
curlopt_maxredirs
: c_int¶
-
const
curlopt_filetime
: c_int¶
-
const
curlopt_telnetoptions
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_TELNETOPTIONS.html
-
const
curlopt_maxconnects
: c_int¶
-
const
curlopt_closepolicy
: c_int¶ This option is deprecated
-
const
curlopt_fresh_connect
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_FRESH_CONNECT.html
-
const
curlopt_forbid_reuse
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_FORBID_REUSE.html
-
const
curlopt_random_file
: c_int¶
-
const
curlopt_egdsocket
: c_int¶
-
const
curlopt_connecttimeout
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_CONNECTTIMEOUT.html
-
const
curlopt_headerfunction
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_HEADERFUNCTION.html
-
const
curlopt_httpget
: c_int¶
-
const
curlopt_ssl_verifyhost
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYHOST.html
-
const
curlopt_ssl_cipher_list
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_SSL_CIPHER_LIST.html
-
const
curlopt_http_version
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_HTTP_VERSION.html
-
const
curlopt_ftp_use_epsv
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_FTP_USE_EPSV.html
-
const
curlopt_sslcerttype
: c_int¶
-
const
curlopt_sslkey
: c_int¶
-
const
curlopt_sslkeytype
: c_int¶
-
const
curlopt_sslengine
: c_int¶
-
const
curlopt_sslengine_default
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_SSLENGINE_DEFAULT.html
-
const
curlopt_dns_use_global_cache
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_DNS_USE_GLOBAL_CACHE.html
-
const
curlopt_dns_cache_timeout
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_DNS_CACHE_TIMEOUT.html
-
const
curlopt_prequote
: c_int¶
-
const
curlopt_debugfunction
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_DEBUGFUNCTION.html
-
const
curlopt_debugdata
: c_int¶
-
const
curlopt_capath
: c_int¶
-
const
curlopt_buffersize
: c_int¶
-
const
curlopt_nosignal
: c_int¶
-
const
curlopt_proxytype
: c_int¶
-
const
curlopt_encoding
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html
-
const
curlopt_private
: c_int¶
-
const
curlopt_http200aliases
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_HTTP200ALIASES.html
-
const
curlopt_unrestricted_auth
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_UNRESTRICTED_AUTH.html
-
const
curlopt_ftp_use_eprt
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_FTP_USE_EPRT.html
-
const
curlopt_httpauth
: c_int¶
-
const
curlopt_ssl_ctx_function
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_SSL_CTX_FUNCTION.html
-
const
curlopt_ssl_ctx_data
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_SSL_CTX_DATA.html
-
const
curlopt_ftp_create_missing_dirs
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_FTP_CREATE_MISSING_DIRS.html
-
const
curlopt_proxyauth
: c_int¶
-
const
curlopt_ftp_response_timeout
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_FTP_RESPONSE_TIMEOUT.html
-
const
curlopt_ipresolve
: c_int¶
-
const
curlopt_maxfilesize
: c_int¶
-
const
curlopt_infilesize_large
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_INFILESIZE_LARGE.html
-
const
curlopt_resume_from_large
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_RESUME_FROM_LARGE.html
-
const
curlopt_maxfilesize_large
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_MAXFILESIZE_LARGE.html
-
const
curlopt_netrc_file
: c_int¶
-
const
curlopt_use_ssl
: c_int¶
-
const
curlopt_postfieldsize_large
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_POSTFIELDSIZE_LARGE.html
-
const
curlopt_tcp_nodelay
: c_int¶
-
const
curlopt_ftpsslauth
: c_int¶
-
const
curlopt_ioctlfunction
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_IOCTLFUNCTION.html
-
const
curlopt_ioctldata
: c_int¶
-
const
curlopt_ftp_account
: c_int¶
-
const
curlopt_ignore_content_length
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_IGNORE_CONTENT_LENGTH.html
-
const
curlopt_ftp_skip_pasv_ip
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_FTP_SKIP_PASV_IP.html
-
const
curlopt_ftp_filemethod
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_FTP_FILEMETHOD.html
-
const
curlopt_localport
: c_int¶
-
const
curlopt_localportrange
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_LOCALPORTRANGE.html
-
const
curlopt_connect_only
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_CONNECT_ONLY.html
-
const
curlopt_conv_from_network_function
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_CONV_FROM_NETWORK_FUNCTION.html
-
const
curlopt_conv_to_network_function
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_CONV_TO_NETWORK_FUNCTION.html
-
const
curlopt_conv_from_utf8_function
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_CONV_FROM_UTF8_FUNCTION.html
-
const
curlopt_max_send_speed_large
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_MAX_SEND_SPEED_LARGE.html
-
const
curlopt_max_recv_speed_large
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_MAX_RECV_SPEED_LARGE.html
-
const
curlopt_ftp_alternative_to_user
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_FTP_ALTERNATIVE_TO_USER.html
-
const
curlopt_sockoptfunction
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_SOCKOPTFUNCTION.html
-
const
curlopt_sockoptdata
: c_int¶
-
const
curlopt_ssl_sessionid_cache
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_SSL_SESSIONID_CACHE.html
-
const
curlopt_ssh_auth_types
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_SSH_AUTH_TYPES.html
-
const
curlopt_ssh_public_keyfile
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_SSH_PUBLIC_KEYFILE.html
-
const
curlopt_ssh_private_keyfile
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_SSH_PRIVATE_KEYFILE.html
-
const
curlopt_ftp_ssl_ccc
: c_int¶
-
const
curlopt_timeout_ms
: c_int¶
-
const
curlopt_connecttimeout_ms
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_CONNECTTIMEOUT_MS.html
-
const
curlopt_http_transfer_decoding
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_HTTP_TRANSFER_DECODING.html
-
const
curlopt_http_content_decoding
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_HTTP_CONTENT_DECODING.html
-
const
curlopt_new_file_perms
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_NEW_FILE_PERMS.html
-
const
curlopt_new_directory_perms
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_NEW_DIRECTORY_PERMS.html
-
const
curlopt_postredir
: c_int¶
-
const
curlopt_ssh_host_public_key_md5
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_SSH_HOST_PUBLIC_KEY_MD5.html
-
const
curlopt_opensocketfunction
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_OPENSOCKETFUNCTION.html
-
const
curlopt_opensocketdata
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_OPENSOCKETDATA.html
-
const
curlopt_copypostfields
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_COPYPOSTFIELDS.html
-
const
curlopt_proxy_transfer_mode
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_PROXY_TRANSFER_MODE.html
-
const
curlopt_seekfunction
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_SEEKFUNCTION.html
-
const
curlopt_seekdata
: c_int¶
-
const
curlopt_crlfile
: c_int¶
-
const
curlopt_issuercert
: c_int¶
-
const
curlopt_address_scope
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_ADDRESS_SCOPE.html
-
const
curlopt_certinfo
: c_int¶
-
const
curlopt_username
: c_int¶
-
const
curlopt_password
: c_int¶
-
const
curlopt_proxyusername
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_PROXYUSERNAME.html
-
const
curlopt_proxypassword
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_PROXYPASSWORD.html
-
const
curlopt_noproxy
: c_int¶
-
const
curlopt_tftp_blksize
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_TFTP_BLKSIZE.html
-
const
curlopt_socks5_gssapi_service
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_SOCKS5_GSSAPI_SERVICE.html
-
const
curlopt_socks5_gssapi_nec
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_SOCKS5_GSSAPI_NEC.html
-
const
curlopt_protocols
: c_int¶
-
const
curlopt_redir_protocols
: c_int¶ See https://curl.haxx.se/libcurl/c/CURLOPT_REDIR_PROTOCOLS.html
-
const
curlopt_lastentry
: c_int¶ All other curlopt values will be less than this one
-
const
curlopt_mail_from
: c_int¶
-
const
curlopt_mail_rcpt
: c_int¶
-
const
curlopt_mail_auth
: c_int¶