In this article, I will be analysing the main aspects of how accessory_java (main repository) manages collections. The code of another Java library based on similar ideas, ib (main repository), is also used as a supporting resource.
I started developing accessory_java while working on a personal project. It was meant to be a container of generic resources which I could eventually use in other applications. Basically, a library including not just useful methods and variables, but also a solid starting point to face virtually any project in Java. Or, in other words, a way to adapt Java to my programming approach, to help me easily overcome the language peculiarities. That initial code has kept growing and evolving until reaching the current stage, which I consider mature, comprehensive and reliable enough.
Main features of accessory_java:
- Built from scratch, by relying on stable functionalities and with the minimum amount of external dependencies possible. At the moment, there is only one external dependency: the MySQL connector. And it can even be ignored in case of not using the database (DB) resources.
- Including virtually no comments or documentation, but with an overall clear structure and quite rigid rules and conventions which are applied in a very systematic and consistent way. Actually, this article and potential future ones might be gradually compensating the aforementioned lacks.
- Main priorities (ordered): friendliness, safety (e.g., default validity checks everywhere or no uncaught exceptions), efficiency/speed, scalability/adaptability.
- Theoretically technology agnostic, although currently mostly focused on MySQL and Linux.
Before using any of the resources of this library, one of the start overloads in the accessory._ini class has to be called, as shown in the code below.
String app_name = samples.APP;
boolean includes_legacy = false; //When true, it looks for a package called "legacy" which might
//be missing anyway.
//------
//dbs_setup includes the setup to be used by default in all the DBs.
//It can be populated by using one of the parent_ini_db.get_setup_vals methods or via adding valid
//types.CONFIG_DB_SETUP_[*]-value pairs. Any of those values can be overwritten by the specific setups
//in the _ini_db class, which are included when adding a new DB through the populate_db method called
//from populate_all_dbs.
String db_name = samples.PERFECT_NAME;
String username = samples.PERFECT_USERNAME;
String password = samples.PERFECT_PASSWORD;
//--- These two variables work together: if encrypt is true, user has to be non-null and the DB
//credentials will be retrieved from the corresponding encrypted file.
boolean encrypt = true;
String user = samples.USER; //This is the crypto ID associated with the given encryption information,
//which can be generated by calling the method db.encrypt_credentials.
//---
//If encrypt is true, a set of encrypted credentials located in a specific file is expected.
//These credentials can be generated by calling db.encrypt_credentials/db.update_credentials.
//But that call or other related actions (e.g., changing the directory for credentials) can't
//be performed before _ini.start is called. That is, if delay_encryption is set to false,
//dbs_setup would be updated with previously-generated credentials. Alternatively, delay_encryption
//can be true, dbs_setup would include a meaningless placeholder and the credentials would be updated
//after _ini.start is called.
boolean delay_encryption = true;
HashMap<String, Object> dbs_setup = null;
if (samples.USE_DB)
{
String setup = null; //This variable can be null (i.e., default setup) or equal to the given DB
//(i.e., corresponding types.CONFIG_[*] constant).
String host = null; //A null value indicates that the default host will be used
//(e.g., "localhost" in MySQL).
if (encrypt && !delay_encryption)
{
dbs_setup = parent_ini_db.get_setup_vals(db_name, setup, user, host, encrypt);
}
else dbs_setup = parent_ini_db.get_setup_vals(db_name, setup, username, password, host);
}
//------
_ini.start(app_name, includes_legacy, dbs_setup);
if (encrypt && delay_encryption)
{
paths.update_dir(paths.DIR_CRYPTO, samples.PERFECT_LOCATION);
paths.update_dir(paths.DIR_CREDENTIALS, samples.PERFECT_LOCATION);
db.update_credentials(db_crypto.SOURCE, user, username, password);
}public abstract class samples
{
public static final String APP = "whatevs";
public static final String ID = "whichevs";
public static final String USER = "whoevs";
public static final String PLACEHOLDER = "wherevs";
//Set this variable to true only if there is a valid DB setup in place
//(i.e., MySQL connector, valid DB name and valid credentials).
public static final boolean USE_DB = false;
//--- I would input my own values where these constants are used if I were you.
public static final String PERFECT_NAME = PLACEHOLDER;
public static final String PERFECT_LOCATION = PLACEHOLDER;
public static final String PERFECT_USERNAME = PLACEHOLDER;
public static final String PERFECT_PASSWORD = PLACEHOLDER;
//---
public static void print_error(String sample_id_) { print_message(sample_id_, null, false); }
public static void print_message(String sample_id_, String message_, boolean is_ok_)
{
String message = sample_id_ + misc.SEPARATOR_CONTENT;
if (is_ok_) message += message_;
else message += "ERROR";
generic.to_screen(message);
}
public static void print_messages(String sample_id_, String[] messages_)
{
String message = sample_id_;
for (String message2: messages_) { message += misc.SEPARATOR_CONTENT + message2; }
generic.to_screen(message);
}
public static void print_end() { generic.to_screen("sample code completed successfully"); }
}Previous articles about accessory_java:
- Encryption -- accessory_java.
- Context, conventions, structure, evolution -- accessory_java.
- Friendliness, safety, efficiency and adaptability -- accessory_java.
- Types and classes -- accessory_java.
A Java library easing the automated communication with Interactive Brokers (IB). It relies on and emulates the defining ideas of accessory_java. An analysis of this library is beyond the scope of the present article and its code will be merely used to support the explanations about accessory_java.
Collections are being collectively referred in accessory_java as "arrays", mostly because this is the name of the main class dealing with them. I came up with this name on the go and haven't ever thought about modifying it. This designation is indeed slightly confusing and, sometimes, it might not be completely clear whether I am referring to collections in general or to arrays in the strict sense (e.g., String[] array). Some people would probably think that a change is needed and they might not be wrong. Their voices will be heard and they will surely have all my moral support, even some of my thoughts and prayers. But we all have to be ready for the worst case scenario, to learn to live with this situation. In other words, I will be using "arrays" to generically refer to collections, unless clearly indicated otherwise, for example via expressions like "simple arrays" or "pure arrays". Good understanders shouldn't have any problem getting it right every time.
static final Class<?>[] populate_all_classes()
{
ArrayList<Class<?>> output = new ArrayList<Class<?>>();
output.add(HashMap.class);
output.add(ArrayList.class);
output.addAll(new ArrayList<Class<?>>(Arrays.asList(get_all_classes_array())));
output.addAll(new ArrayList<Class<?>>(Arrays.asList(get_all_classes_small())));
return output.toArray(new Class<?>[output.size()]);
}
static Class<?>[] populate_all_classes_small()
{
return new Class<?>[]
{
double[].class, long[].class, int[].class, boolean[].class, byte[].class, char[].class
};
}
static Class<?>[] populate_all_classes_array()
{
return new Class<?>[]
{
Array.class, String[].class, Boolean[].class, Integer[].class, Long[].class, Double[].class,
Byte[].class, Character[].class, Class[].class, Method[].class, Exception[].class,
LocalTime[].class, LocalDateTime[].class, size[].class, data[].class, db_field[].class,
db_where[].class, db_order[].class
};
}The duality "small"/"big" arrays represents another naming peculiarity of these libraries where the ideas in the previous paragraph are also applicable. Java calls "primitive" what I call "small", by evidently reserving "big" for the Object-inherited version (e.g., Double vs. double). This is also a practical example of context helping to understand the distinction discussed in the previous paragraph. When talking about "small"/"big" arrays, I will always be referring to simple arrays because such a differentiation wouldn't make any sense with the other supported collections. Either way, I will stop using quotes for these peculiar designations, now and in future articles. Such references might be unorthodox in general, but not within the context of the current article and these libraries, where small arrays will always be "small" "arrays".
public static boolean is_array(double[] input_) { return true; }
public static boolean is_array(long[] input_) { return true; }
public static boolean is_array(int[] input_) { return true; }
public static boolean is_array(boolean[] input_) { return true; }
public static boolean is_array(byte[] input_) { return true; }
public static boolean is_array(char[] input_) { return true; }
public static boolean is_array(Object input_)
{
return is_common(input_, arrays.get_all_classes(), false);
}For practical reasons, accessory_java only supports the following types of collections:
HashMap.ArrayList.- 1D arrays are fully supported and multidimensional arrays are just partially supported. For arrays and in general, accessory_java only supports the following small types:
double,long,int,boolean,byteandchar.
The main method determining the class of the given variable/constant (i.e., accessory.generic.get_class) considers all the HashMaps identical to each other, as well as all the ArrayLists. However, accessory.arrays also contains various methods allowing to extract the constituent classes of non-empty instances for all the supported collections.
static Class<?>[] get_classes_items(Object input_, boolean check_input_)
{
Class<?> type = generic.get_class(input_);
if (!generic.are_equal(type, HashMap.class)) return null;
return new Class<?>[]
{
get_class_key_val(input_, true, false, check_input_),
get_class_key_val(input_, false, false, check_input_)
};
}
@SuppressWarnings("unchecked")
static <x> Class<?> get_class_items(Object input_, boolean check_input_)
{
Class<?> output = null;
Class<?> type = generic.get_class(input_);
if (!generic.is_array(type) || generic.are_equal(type, HashMap.class)) return output;
if (type.equals(double[].class)) return get_class_items((double[])input_, check_input_);
else if (type.equals(long[].class)) return get_class_items((long[])input_, check_input_);
else if (type.equals(int[].class)) return get_class_items((int[])input_, check_input_);
else if (type.equals(boolean[].class)) return get_class_items((boolean[])input_, check_input_);
else if (type.equals(byte[].class)) return get_class_items((byte[])input_, check_input_);
else if (type.equals(char[].class)) return get_class_items((char[])input_, check_input_);
else
{
ArrayList<x> arraylist = null;
x[] array = null;
boolean is_arraylist = generic.are_equal(type, ArrayList.class);
if (is_arraylist) arraylist = (ArrayList<x>)input_;
else array = (x[])input_;
for (int i = 0; i < (is_arraylist ? arraylist.size() : array.length); i++)
{
output = get_class_item((is_arraylist ? arraylist.get(i) : array[i]), output);
if (generic.are_equal(output, Object.class)) break;
}
}
return output;
}Simple arrays are equivalent to others having the same class or the corresponding small/big counterpart. That is, accessory.generic.get_class returns different values for int[] array1 = new int[] { 1, 2 }; and Integer[] array2 = new Integer[] { 1, 2 };, but both outputs are assumed to be identical according to accessory.generic.are_equal.
static HashMap<Class<?>, Class<?>[]> populate_all_class_equivalents()
{
HashMap<Class<?>, Class<?>[]> output = new HashMap<Class<?>, Class<?>[]>();
output.put(Double.class, new Class<?>[] { double.class });
output.put(Long.class, new Class<?>[] { long.class });
output.put(Integer.class, new Class<?>[] { int.class });
output.put(Boolean.class, new Class<?>[] { boolean.class });
output.put(Byte.class, new Class<?>[] { byte.class });
output.put(Character.class, new Class<?>[] { char.class });
output.put(Double[].class, new Class<?>[] { double[].class });
output.put(Long[].class, new Class<?>[] { long[].class });
output.put(Integer[].class, new Class<?>[] { int[].class });
output.put(Boolean[].class, new Class<?>[] { boolean[].class });
output.put(Byte[].class, new Class<?>[] { byte[].class });
output.put(Character[].class, new Class<?>[] { char[].class });
output.put(Array.class, new Class<?>[] { Object[].class });
return output;
}An important idea to bear in mind is that I see programming languages as tools, as the means which I have to maximise to accomplish the goal of creating a piece of software meeting the given requirements. I do feel more comfortable working with certain languages/features and even tend to rely on similar approaches almost regardless of the used technology. Although this should be seen as a consequence of my specific expertise which, in many cases, has been mostly defined by external factors on which I had no control. In other words, I rarely see big problems with a programming language, not up to the point of not using it because of them. If the choice is made by others, I would simply try to get the most from that language by seeing any problem as part of my normal work, which is basically solving problems. Long story short, what I am writing here or in subsequent sections shouldn't be seen as critics or complaints. They are just descriptions of specific requirements, the conditions justifying certain parts of the code, even the main purpose of these libraries: adapting Java to how I program, helping me do my work better and more efficiently.
Another important idea is that these libraries aren't about deeply understanding and optimising the Java language. They are expected to be reliable and efficient enough, to really maximise what is available, although only at a relatively superficial level, by always fully accepting their pragmatic and instrumental essence. Some of the discussed problems, for example, provoked an almost immediate implementation, in some cases with virtually no research or a profound understanding of the given issue. In my opinion, such an understanding isn't even recommendable in some scenarios, almost an unnecessary waste of time. It doesn't need to be a real problem either, not for everyone and in every situation. Something going against my intuitive expectations, which could have grown by using languages other than Java, is all I need. I am experienced enough to have a pretty good grasp of what I should expect or, better, of what I want to get. These libraries are about helping me with my work, and avoiding exceptional scenarios or doubts seems a hell of a help. Some of the discussed issues might not be considered problems by many people or there might be good justifications for their existence or Java might already offer specific solutions which I am ignoring, but nothing of that would truly matter to me anyway.
The referred small/big duality is a pretty important peculiarity, mainly by bearing in mind the intended equivalence. The default compatibility of Java for small/big types of non-arrays isn't exactly perfect, although it does meet most of my expectations. With arrays, the story changes completely. For example, small arrays are compatible with Object but not with Object[], and this is a big problem. Object array = new int[0]; is valid and Object[] array2 = new int[0]; is not. All the big arrays inherit from Object[], where all the generic parameters (e.g., <x> my_method(x[] param_)) are also included. That's why and additionally to a method considering their classes identical, small/big conversions are also needed in order to accomplish the intended small/big compatibility.
public static char[] to_small(Character[] input_)
{
int tot = get_size(input_);
if (tot < 1) return null;
char[] output = new char[tot];
for (int i = 0; i < tot; i++) { output[i] = (input_[i] != null ? input_[i] : DEFAULT_CHAR); }
return output;
}
public static double[][] to_small(Double[][] input_)
{
int tot = get_size(input_);
if (tot < 1) return null;
double[][] output = new double[tot][];
for (int i = 0; i < tot; i++)
{
if (input_[i] == null) continue;
int tot2 = input_[i].length;
output[i] = new double[tot2];
for (int i2 = 0; i2 < tot2; i2++)
{
output[i][i2] = (input_[i][i2] != null ? output[i][i2] : DEFAULT_DECIMAL);
}
}
return output;
}The fact that the generic parameters can only be used with big arrays explains an important proportion of the accessory.arrays code. Specific overloads for all the supported small types are required to accomplish the expected friendliness. And this issue does represent a practical limitation for the growth of that class. A fairly big amount of code is required to implement virtually any functionality. Adding new methods to accessory.arrays_quick is much easier, although the conditions there are much more specific and restrictive.
public static String to_string
(
Object input_, String sep_items_, String sep_keyval_, String[] keys_ignore_
)
{
return
(
generic.are_equal(generic.get_class(input_), HashMap.class) ?
to_string(input_, sep_items_, sep_keyval_, keys_ignore_, false, true) : strings.DEFAULT
);
}
public static String to_string(double[] input_, String separator_)
{
return to_string(to_big(input_), separator_, true);
}
public static String to_string(long[] input_, String separator_)
{
return to_string(to_big(input_), separator_, true);
}
public static String to_string(int[] input_, String separator_)
{
return to_string(to_big(input_), separator_, true);
}
public static String to_string(boolean[] input_, String separator_)
{
return to_string(to_big(input_), separator_, true);
}
public static String to_string(byte[] input_, String separator_)
{
return strings.from_bytes_base64(input_);
}
public static String to_string(char[] input_, String separator_)
{
return to_string(to_big(input_), separator_, true);
}
public static <x> String to_string(x[] input_, String separator_)
{
return to_string(input_, separator_, true);
}
public static <x> String to_string(ArrayList<x> input_, String separator_)
{
return to_string(input_, separator_, true);
}public static int[] get_new(int[] input_) { return Arrays.copyOfRange(input_, 0, input_.length); }
public static String[] get_new(String[] input_) { return Arrays.copyOfRange(input_, 0, input_.length); }The problems with overloads in the accessory.arrays class aren't limited to small arrays. The generic HashMap or ArrayList parameters are less problematic, but certain issues still need to be addressed. Two generic parameters (e.g., <x, y> my_method(HashMap<x, y> param_)) are needed to account for all the HashMap scenarios. Under most conditions, such a setup works reasonably well, although both parameters aren't always considered different. It is impossible, for example, to put <x> my_method(HashMap<x, x> param_) and <x, y> my_method(HashMap<x, y> param_) in the same class because the compiler (Eclipse) assumes that they are identical. Such eventuality forces either a reliance on two methods with different names (e.g., by appending "_xx" and "_xy") or the addition of an overload with an Object parameter. This last option provokes further problems on account of how Java deals with these scenarios. If both my_method(int[] param_) and my_method(Object param_) exist in the same class, int[] array1 = new int[] { 1, 2 }; would reach the first one and Object array2 = new int[] { 1, 2 }; the second. This means that my_method(Object param_) also has to be able to internally deal with int[] arguments. Another issue with these Object overloads is that the determination of the parameter class, a basic requirement for some of these methods, is impossible for null inputs.
@SuppressWarnings("unchecked")
public static <x> String to_string(Object input_, String separator_, boolean include_brackets_)
{
String output = strings.DEFAULT;
Class<?> type = generic.get_class(input_);
if (!generic.is_array(type)) return output;
if (type.equals(double[].class)) return to_string((double[])input_, separator_);
else if (type.equals(long[].class)) return to_string((long[])input_, separator_);
else if (type.equals(int[].class)) return to_string((int[])input_, separator_);
else if (type.equals(boolean[].class)) return to_string((boolean[])input_, separator_);
else if (type.equals(byte[].class)) return to_string((byte[])input_, separator_);
else if (type.equals(char[].class)) return to_string((char[])input_, separator_);
boolean is_array = generic.is_array_class(type);
boolean is_arraylist = generic.are_equal(type, ArrayList.class);
if (!is_array && !is_arraylist) return output;
output = "";
String separator = (strings.is_ok(separator_) ? separator_ : misc.SEPARATOR_ITEM);
boolean first_time = true;
if (generic.are_equal(type, Byte[].class)) output = strings.from_bytes_base64((Byte[])input_);
else if (is_array)
{
for (x item: (x[])input_)
{
if (first_time) first_time = false;
else output += separator;
output += to_string_val(item);
}
}
else if (is_arraylist)
{
for (x item: (ArrayList<x>)input_)
{
if (first_time) first_time = false;
else output += separator;
output += to_string_val(item);
}
}
if (include_brackets_ && strings.is_ok(output))
{
output = misc.BRACKET_MAIN_OPEN + output + misc.BRACKET_MAIN_CLOSE;
}
return output;
}@SuppressWarnings("unchecked")
public static <x> HashMap<x, x> get_new_hashmap_xx(HashMap<x, x> input_)
{
return (HashMap<x, x>)get_new_hashmap(input_, false);
}
@SuppressWarnings("unchecked")
public static <x, y> HashMap<x, y> get_new_hashmap_xy(HashMap<x, y> input_)
{
return (HashMap<x, y>)get_new_hashmap(input_, true);
}With simple arrays, it seems clear that int[][] or Double[][][] are multidimensional. Cases like HashMap<String, String>[] might not be that straightforward, although pretty evident too. With other collections like HashMaps or ArrayLists, that label seems intuitively less applicable. But there are really no essential differences between a pure array and a HashMap or an ArrayList, and, consequently, similar ideas should be applied to all of them. That is, ArrayList<ArrayList<String>> is multidimensional, exactly the same than HashMap<String, Double[]> or HashMap<Integer, HashMap<Integer, HashMap<Integer, Integer>>>.
@SuppressWarnings("unchecked")
private static volatile ArrayList<Double>[] _flus = new ArrayList[SIZE_GLOBALS_FLUS];
@SuppressWarnings("unchecked")
private static volatile ArrayList<Double>[] _flus2_minus = new ArrayList[SIZE_GLOBALS_FLUS];
@SuppressWarnings("unchecked")
private static volatile ArrayList<Double>[] _flus2_plus = new ArrayList[SIZE_GLOBALS_FLUS];The support for multidimensional HashMaps and ArrayLists is already included in the 1D overloads. <x> my_method(ArrayList<x> param_) accepts ArrayList<String> array1 and also ArrayList<ArrayList<ArrayList<Integer>>> array2. <x> my_method(x[] param_) gracefully dealing with inputs like String[][][] array seems less intuitive. And the fact that the main inbuilt Java class for pure arrays, java.utils.Arrays, seamlessly supports multidimensionality appears still more counterintuitive. Either way, multidimensional simple arrays are well supported by the default 1D implementation too, at least theoretically, not so much from the practical perspective of these libraries. Firstly, it is more natural to me to rely on multidimensional HashMaps and, for this reason, the internal algorithms of the methods dealing with them are prepared for these scenarios, unlike what happens in the case of pure arrays or even ArrayLists. Secondly, dimensionality is indeed relevant for small-array overloads and, consequently, specific implementations accounting for all the potential scenarios would be required anyway to guarantee the small/big compatibility. In summary, accessory.arrays supports multidimensional simple arrays only partially and under very specific conditions. The default overloads for pure arrays (i.e., the ones including x[] or Object parameters but not multidimensional ones like x[][]) are expected to be exclusively used with 1D inputs.
public static Byte[][] to_big(byte[][] input_)
{
int tot = get_size(input_);
if (tot < 1) return null;
Byte[][] output = new Byte[tot][];
for (int i = 0; i < tot; i++)
{
if (input_[i] == null) continue;
int tot2 = input_[i].length;
output[i] = new Byte[tot2];
for (int i2 = 0; i2 < tot2; i2++) output[i][i2] = (Byte)input_[i][i2];
}
return output;
}
public static Character[][] to_big(char[][] input_)
{
int tot = get_size(input_);
if (tot < 1) return null;
Character[][] output = new Character[tot][];
for (int i = 0; i < tot; i++)
{
if (input_[i] == null) continue;
int tot2 = input_[i].length;
output[i] = new Character[tot2];
for (int i2 = 0; i2 < tot2; i2++) output[i][i2] = (Character)input_[i][i2];
}
return output;
}public static double[] get_1d(double[][] array_, int i_)
{
return Arrays.copyOfRange(array_[i_], 0, array_[i_].length);
}I like relying on HashMaps (or on the equivalent type in other languages) quite a lot, even though they are anything but efficient and I do think that my overall programming approach is pretty efficiency-focused. Also, when using them, multidimensionality is certainly an option for me, not so much with ArrayLists or simple arrays. In general, relying on multidimensional collections is the easiest, friendliest and most inefficient alternative. Replacing a 2D array with two 1D ones, for example, is likely to make the given code more efficient and easier to further optimise. But the former is usually less difficult to implement and, in some cases, not even much less efficient. Nonetheless, my usual approach is to firstly focus on HashMaps/ArrayLists, ideally 1D but by easily accepting multidimensionality, and, at a later stage and if proven required, reducing dimensions or even moving to pure arrays.
Actually, I made a recent modification in accessory.db which is pretty descriptive of my ideas on this front. The original version included various multidimensional HashMaps.
static HashMap<String, HashMap<String, db_field>> SOURCES =
new HashMap<String, HashMap<String, db_field>>();
private static HashMap<String, HashMap<String, Object>> SOURCE_SETUPS =
new HashMap<String, HashMap<String, Object>>();
private static HashMap<String, Boolean> SOURCE_DEFAULTS = new HashMap<String, Boolean>();
private static HashMap<String, String> DB_SETUPS = new HashMap<String, String>();After the first optimisation, there were still three 2D arrays remaining, as shown in the code below.
static String[] SOURCES = new String[0];
static String[] SETUP_IDS = new String[0];
static String INSTANCE = null;
static String _cur_source = strings.DEFAULT;
static String _cur_type = DEFAULT_TYPE;
private static db_field[][] SOURCE_FIELDS = new db_field[0][0];
private static String[][] SOURCE_FIELDS2 = new String[0][0];
private static Object[][] SOURCE_SETUPS = new Object[0][0];
private static boolean[] SOURCE_DEFAULTS = new boolean[0];And, finally, I came up with the current version, which only includes one small 2D array which isn't even used too often.
static String[] SETUP_IDS = new String[0];
static String[] SOURCES = new String[0];
static String[] FIELDS = new String[0];
static String INSTANCE = null;
static String _cur_source = strings.DEFAULT;
static String _cur_type = DEFAULT_TYPE;
private static int[] SOURCES0 = new int[0];
private static db_field[] FIELDS0 = new db_field[0];
private static Object[][] SETUPS = new Object[0][0];
private static boolean[] DEFAULTS = new boolean[0];The performance of the given piece of software is usually inversely related to its memory usage: the more memory it uses, the more likely that it will be slower, either directly by itself or indirectly by suffering the consequences of its overuse having affected other computer resources. It seems clear that a lower memory consumption is, in general, a positive outcome. Although, in some cases and mainly for short periods of time, a higher memory use can lead to better performance within the short term, what might be translated into an overall more efficient approach in the long run. In any case, an extensive reliance on arrays, especially on ones which are big and long-lived enough, has undoubtedly a relevant impact on both performance and efficiency. Global arrays represent an effective way to tune memory usage and code speed in these libraries. The algorithmic, local resources at the method level can easily be sped up via caching, either quickly through memory or slowly by using DB/files. And the way in which this quick caching is performed, the type and characteristics of the used arrays, has therefore an important role on how the corresponding application behaves.
private static HashMap<String, HashMap<String, Object>> _info =
new HashMap<String, HashMap<String, Object>>();static volatile String[] _symbols = new String[SIZE_GLOBALS];
@SuppressWarnings("unchecked")
static volatile HashMap<String, String>[] _vals = (HashMap<String, String>[])new HashMap[SIZE_GLOBALS];
static volatile int[] _fields_ib = null;Evidently, local arrays could also be very influential memory- and performance-wise. A method carrying out demanding tasks can have a dramatic impact on memory (e.g., many big local arrays) or CPU (e.g., quick loops running for long periods). And such a setup can surely be useful, almost required in some cases, but it should be considered an error in the context of these libraries. The global, class-level side is, hence, the only relevant scenario here. And my tendency to use multidimensional HashMaps, also at the global level, is an important aspect to bear in mind. Out of all the collections supported by accessory_java, global HashMaps are the most memory-intensive ones, especially their multidimensional versions. Thus, reducing their sizes, dimensions or even presence is a good measure to reduce memory usage and, in most cases, to also improve the performance of specific parts.
private static ArrayList<String> ROOTS = null;
private static HashMap<String, String>[] TYPES_KEYS = null;
private static String[] ROOTS_TO_IGNORE = null;After having built a very friendly code, I have begun to worry about reducing its overall memory footprint, mostly by focusing on making global collections more efficient. Under ideal conditions and as a rule of thumb, I would say that acceptably efficient sizes for 1D collections are: 250 for HashMaps, 500 for ArrayLists and 1000-2000 for pure arrays. The small arrays are, by definition, more efficient than their big counterparts. The multidimensional versions should be avoided when possible; valid exceptions are 2D simple arrays and, when being small or short-lived enough, higher dimensionality or other collection types.
static String[] _cols = new String[0];
private static ArrayList<String> QUICKER_MYSQL = null;HashMaps and ArrayLists aren't always a problem from a performance point of view. Their memory overuse is, sometimes, well compensated by the usefulness of their native functionalities. Situations involving multiple additions, deletions or element searches are paradigmatic scenarios. Just something as simple as having to regularly retrieve the index associated with a given value might already be a good reason to ditch simple arrays. This has in fact been the case with these libraries until recently: a general preference for global ArrayLists over simple arrays just to use some of their inbuilt resources (e.g., indexOf or contains). My aforementioned new concerns, together with the support of the accessory.arrays_quick methods, have changed that tendency. However, it isn't always too clear which option is really faster; at least in this context, where their sizes are quite small, otherwise there would be an uncontested winner. Either way, simple arrays will always consume less memory and, when running for long periods/being constantly accessed, will tend to perform better. Descriptive samples of this evolution can be found in ib, for instance, the code which I improved from there to here.
public static boolean value_exists(int[] array_, int value_)
{
return (get_i(array_, value_) != arrays.WRONG_I);
}
public static boolean value_exists(String[] array_, String value_)
{
return (get_i(array_, value_) != arrays.WRONG_I);
}
public static int get_i(int[] array_, int value_)
{
for (int i = 0; i < array_.length; i++)
{
if (array_[i] == value_) return i;
}
return arrays.WRONG_I;
}
public static int get_i(String[] array_, String value_)
{
for (int i = 0; i < array_.length; i++)
{
if (array_[i].equals(value_)) return i;
}
return arrays.WRONG_I;
}
public static int get_i(String[][] array_, int i_, String value_) { return get_i(array_[i_], value_); }As explained above, these are the collections which are defined at the class level, perhaps the resources with the highest impact on memory usage and performance in these libraries.
Java is particularly peculiar when dealing with global collections, especially with the more complex ones. Even inline populations, an option which I tend to use often with other languages, have some issues. With simple arrays (e.g., int[] array = new int[] { 1, 2 };), everything works fine. But, with ArrayLists (e.g., ArrayList<Integer> array = new ArrayList<Integer>() { { add(1); add(2); } };) and HashMaps (e.g., HashMap<Integer, Integer> array = new HashMap<Integer, Integer>() { { put(1, 2); } };), I have witnessed too much irregularity. As already discussed, I don't want to (or think that really should) dive deep into the reasons why counterintuitive-to-me behaviours happen. All I want is to be able to easily rely on approaches and resources facilitating my work. Virtually anything can be done in alternative ways and I am experienced enough to feel comfortable under a wide variety of different conditions. Long story short: one single occurrence of an unexpected outcome, fully validated and confirmed, could be more than enough for me to stop using that option for good. Avoiding inline populations is easy and the HashMaps/ArrayLists ones have proven to not be dependable, mainly on the global front, not as per my expectations, which pure arrays fulfill perfectly. Initialisation (e.g., int[] array = new int[2];, ArrayList<Integer> array = new ArrayList<Integer>(); or new HashMap<Integer, Integer>();), on the other hand, is indeed very reliable.
private static volatile ArrayList<Integer> _out_ints = new ArrayList<Integer>();
private static volatile ArrayList<String> _out_strings = new ArrayList<String>();
private static volatile ArrayList<Double> _out_decimals = new ArrayList<Double>();A very important aspect of global collections is their population/assignation of values, mainly the exact moment when such an event occurs. This timing is crucial at the library's startup, where multiple interdependent resources have to be populated in a specific order. And this is precisely the part where Java is most peculiar with global arrays. When the class is loaded for the first time, the exact moment when each element defined at the class level is reached and its inline actions are performed can't be determined as accurately as I would like. Issues like the defining features of the array (e.g., including the final keyword or not) or the way in which the class is being loaded (e.g., by making a call to a method or to another global resource) seem irrelevant. The resulting behavior appears to be conditioned by the essence of the given resource, by the way in which their inner workings are synchronised with the ones triggered when the class is being loaded. That is, a HashMap being assigned to anything other than a blank initialisation or null might be reached in the right moment, but the elements assumed to be added on the spot might still not be there before the next global resource is reached.
public static Class<?>[] ARRAYS_CLASSES_SMALL = null;
public static Class<?>[] ARRAYS_CLASSES_ARRAY = null;
public static Class<?>[] ARRAYS_CLASSES_NUMERIC = null;
public static Class<?>[] ARRAYS_CLASSES = null;
public static HashMap<Boolean, String[]> STRINGS_BOOLEANS = null;
public static char[] STRINGS_EXPS = null;
public static Class<?>[] NUMBERS_CLASSES = null;
public static Class<?>[] GENERIC_CLASSES = null;
public static HashMap<Class<?>, Class<?>[]> GENERIC_CLASSES_EQUIVALENTS = null;
public static String[] GENERIC_DEFAULT_METHOD_NAMES = null;I have also observed those peculiar behaviours of global arrays, especially HashMaps, when the values are added indirectly (i.e., from a local method). These scenarios are much more exceptional and seem to be provoked by complex class loading setups. In the part of the startup where emulation is enabled, for example, the classes are loaded by making a dynamic call to the specific method, by using Java reflection via accessory.generic.call_static_method. And, in some cases, I have seen global HashMaps behaving counterintuitively when that dynamic setup was used to perform "crossed" actions (e.g., after the accessory startup resources have been loaded, adding values to a global HashMap in accessory from a dynamically-loaded accessory_ib startup class by using an accessory method). Similar ideas should be applied to the default asynchronous behaviour (i.e., without locks) of HashMaps: unexpected timings might occur.
private void populate_all_internal_other_class(String package_, String class_)
{
String method0 = "populate";
String name = package_ + "." + class_;
Class<?> class1 = generic.get_class_from_name(name);
if (class1 == null) return;
generic.ignore_errors(true);
Class<?>[] params = null;
Object[] args = null;
if (class_.equals("_ini_db"))
{
params = new Class<?>[] { HashMap.class, String[].class };
args = new Object[]
{
new HashMap<String, Object>(DBS_SETUP), (String[])arrays.get_new(TYPES_TO_IGNORE)
};
}
else if (class_.equals("_keys"))
{
params = new Class<?>[] { String[].class };
args = new Object[] { (String[])arrays.get_new(TYPES_TO_IGNORE) };
}
Method method = null;
int max = 2;
int count = 0;
while (count < max)
{
method = generic.get_method(class1, method0, params);
if (method != null) break;
count++;
if (class_.equals("_ini_db"))
{
if (count == 1)
{
params = new Class<?>[]
{
String.class, String.class, String.class, String.class, boolean.class
};
args = new Object[]
{
get_setup_val(_types.CONFIG_DB_SETUP_CREDENTIALS_USER),
get_setup_val(_types.CONFIG_DB_SETUP_CREDENTIALS_USERNAME),
get_setup_val(_types.CONFIG_DB_SETUP_CREDENTIALS_PASSWORD),
get_setup_val(_types.CONFIG_DB_SETUP_HOST),
get_setup_val(_types.CONFIG_DB_SETUP_CREDENTIALS_ENCRYPTED)
};
}
else if (count == 2)
{
params = null;
args = null;
}
else return;
}
else if (class_.equals("_keys"))
{
if (count == 1)
{
params = null;
args = null;
}
}
else return;
}
generic.call_static_method(method, args);
generic.ignore_errors_persistent_end();
}On top of the generic conventions used in these libraries, there are some specific ones for global arrays, a direct consequence of the peculiarities discussed in the previous subsection. Namely:
- All the global collections have to be assigned to a certain value, but never to a population (inline or via method) or to another array. Inline population of pure arrays (e.g.,
double[] array = new double[] { 1.0, 2.0 };) is the only acceptable exception. Global arrays can be assigned tonullor to a blank initialisation (e.g.,HashMap<String, String> array = new HashMap<String, String>();orbyte[] array = new byte[10];). - Global arrays will always get their values from local methods, except for simple arrays being populated inline.
static HashMap<Boolean, String[]> populate_all_booleans()
{
HashMap<Boolean, String[]> booleans = new HashMap<Boolean, String[]>();
booleans.put
(
true, new String[]
{
data.TRUE, from_boolean(true), from_number_int(numbers.to_int(true))
}
);
booleans.put
(
false, new String[]
{
data.FALSE, from_boolean(false), from_number_int(numbers.to_int(false))
}
);
return booleans;
}
static char[] populate_all_exps() { return new char[] { 'e', '^' }; }
private static char[] get_all_exps() { return _alls.STRINGS_EXPS; }- Ideally, the local methods referred in the previous point should be called under clearly-defined, deterministic conditions. In any case, they always have to be local to the given library, what includes inherited features (e.g., accessory_ib._ini_db populating
accessory.dbglobal arrays by relying on its db.parent_ini_db inheritance).
protected boolean populate_source
(
String source_, String table_, HashMap<String, Object[]> fields_,
HashMap<String, Object> setup_vals_, boolean includes_default_fields_
)
{
if (!setup_vals_are_ok(setup_vals_)) return false;
String db = config.check_type((String)setup_vals_.get(_types.CONFIG_DB));
String source = config.check_type(source_);
if
(
!strings.is_ok(db) || !strings.is_ok(source) ||
!strings.is_ok(table_) || !arrays.is_ok(fields_)
)
{ return false; }
HashMap<String, db_field> fields = new HashMap<String, db_field>();
for (Entry<String, Object[]> item: fields_.entrySet())
{
String id = item.getKey();
Object[] val = item.getValue();
String col = (String)val[0];
config.update_ini(db, id, col);
db_field field = (db_field)val[1];
fields.put(id, field);
}
if
(
!accessory.db.add_source_ini
(
source, fields, setup_vals_, includes_default_fields_
)
)
{ return false; }
if (!config.update_ini(db, source, table_)) return false;
return true;
}private void perform_actions_after_population()
{
db_common.populate_is_quick_ini();
db_quick.populate_cols_ini();
db_quick.populate_quicker_ini();
}- When possible and advisable, global arrays should be defined and populated by using the common approaches and resources meant for that purpose (e.g., the startup methods discussed in the next subsection).
public void populate_internal()
{
if (_populated) return;
SYNC_GET_OUTS = sync.populate_all_get_outs();
CALLS_KEYS_FUNDS = calls.populate_all_keys_funds();
EXTERNAL_CONTRACTS_SECURITIES = external_ib.contracts.populate_all_securities();
EXTERNAL_DATA_TICKS = external_ib.data.populate_all_ticks();
EXTERNAL_ORDERS_EXEC_SIDES = external_ib.orders.populate_all_exec_sides();
EXTERNAL_ORDERS_ACTIONS = external_ib.orders.populate_all_actions();
EXTERNAL_ORDERS_TYPES = external_ib.orders.populate_all_types();
EXTERNAL_ORDERS_TIFS = external_ib.orders.populate_all_tifs();
EXTERNAL_ORDERS_STATUSES = external_ib.orders.populate_all_statuses();
DB_SOURCES = db_ib.common.populate_all_sources();
DB_SOURCES_NEW = db_ib.common.populate_all_sources_new();
DB_SOURCES_USER = db_ib.common.populate_all_sources_user();
DB_SOURCES_ENABLED = db_ib.common.populate_all_sources_enabled();
DB_SOURCES_ELAPSED = db_ib.common.populate_all_sources_elapsed();
DB_MAX_SIZES_NUMBERS = db_ib.common.populate_all_max_sizes_numbers();
DB_MAX_SIZES_STRINGS = db_ib.common.populate_all_max_sizes_strings();
DB_MAX_VALS = db_ib.common.populate_all_max_vals();
DB_NEGATIVE_NUMBERS = db_ib.common.populate_all_negative_numbers();
_populated = true;
}The accessory classes whose names start with an underscore and the ones inside the accessory_ib package, an emulation of the former, constitute the startup parts of both libraries. All these classes are loaded at the very beginning by following a certain order and are meant, additionally to manage the most basic and important information, to help deal with the peculiarities explained in the previous subsections. That is, the startup parts eminently revolve around global arrays.
accessory._alls/accessory_ib._alls are expected to be used as a centralised repository of generic, simplistic global collections. Note that these classes are being loaded before others which are very important like _ini_db and, consequently, can't deal with most of the features/information of these libraries. These arrays are assumed to be mainly storing global constants (e.g., the ones in the _types classes). The methods interacting with these two classes also follow some sort of convention: the names of the ones storing/retrieving these collections start with "populate_all_"/"get_all_".
protected void populate_internal()
{
ARRAYS_CLASSES_SMALL = arrays.populate_all_classes_small();
ARRAYS_CLASSES_ARRAY = arrays.populate_all_classes_array();
ARRAYS_CLASSES_NUMERIC = arrays.populate_all_classes_numeric();
ARRAYS_CLASSES = arrays.populate_all_classes();
STRINGS_BOOLEANS = strings.populate_all_booleans();
STRINGS_EXPS = strings.populate_all_exps();
NUMBERS_CLASSES = numbers.populate_all_classes();
GENERIC_CLASSES = generic.populate_all_classes();
GENERIC_CLASSES_EQUIVALENTS = generic.populate_all_class_equivalents();
GENERIC_DEFAULT_METHOD_NAMES = generic.populate_all_default_methods();
DATA_CLASSES = data.populate_all_classes();
DATA_COMPATIBLE = data.populate_all_compatible();
DB_SOURCES_INBUILT = db_common.populate_all_sources_inbuilt();
DB_QUERIES_DATA = db.populate_all_queries_data();
DB_SQL_TYPES = db_sql.populate_all_types();
DB_WHERE_OPERANDS = db_where.populate_all_operands();
DB_WHERE_OPERANDS_LIKE = db_where.populate_all_operands_like();
DB_WHERE_OPERANDS_NOT_LIKE = db_where.populate_all_operands_not_like();
DB_WHERE_LINKS = db_where.populate_all_links();
DB_FIELD_TYPES_NO_SIZE = db_field.populate_all_types_no_size();
CONFIG_NOT_UPDATE = config.populate_all_not_update();
CRYPTO_WHATS = crypto.populate_all_whats();
MISC_SUPPORTED_SOUND_EXTENSIONS = misc.populate_all_supported_sound_extensions();
}public static String[] get_all_types_no_size() { return _alls.DB_FIELD_TYPES_NO_SIZE; }
static String[] populate_all_types_no_size()
{
return new String[] { data.TIMESTAMP, data.BOOLEAN, data.TINYINT, data.INT, data.LONG };
}The _types/_keys classes are fully focused on dealing with accessory_java types (i.e., relevant categorised constants representing a cornerstone of these libraries) and their corresponding keys (i.e., human-friendly, pragmatic, meaningful substrings of types). From the perspective of global collections, these classes store all the associated values, either in an absolute, generic way or to accomplish a specific goal like speeding up the retrieval of certain keys. On similar lines, accessory._ini_config/accessory_ib._ini_config update the contents of accessory.config._info.
private boolean load_config_basic()
{
String type = accessory._types.CONFIG_BASIC;
HashMap<String, Object> vals = new HashMap<String, Object>();
vals.put(basic.CONFIG_ID_MAIN, _defaults.ID_MAIN);
return populate(type, null, vals);
}
private boolean load_config_order()
{
String type = _types.CONFIG_ORDERS;
HashMap<String, Object> vals = new HashMap<String, Object>();
vals.put(_order.CONFIG_TIF, _order.DEFAULT_TIF);
vals.put(_order.CONFIG_QUANTITIES_INT, _order.DEFAULT_QUANTITIES_INT);
return populate(type, null, vals);
}Unlike the other startup classes, accessory._ini_db/accessory_ib._ini_db deal with a comprehensive reality, DB management, and affect various other classes. This part will be discussed in detail in future articles. For the time being, it is enough to know that the main global arrays are located in accessory.db and in other accessory classes like accessory.db_quick. These are the most complex, heaviest global arrays which, additionally, contain essential information expected to be accessed very quickly. These global collections including all the DB information are the ones with the highest impact on performance and memory consumption, in general (because of their important sizes) and also because of essentially defining the way in which the DB interactions occur.
static void populate_cols_ini()
{
int tot = db.SOURCES.length;
if (tot == 0) return;
for (String source: db.SOURCES) populate_cols(source, db.get_source_fields(source));
}
private static void populate_cols(String source_, HashMap<String, db_field> fields_)
{
if (!arrays.is_ok(fields_)) return;
for (Entry<String, db_field> item: fields_.entrySet())
{
String field = item.getKey();
_cols = arrays_quick.add(_cols, db.get_field_i(source_, field), db.get_col(source_, field));
}
}@SuppressWarnings("unchecked")
protected boolean populate_all_dbs(HashMap<String, Object> dbs_setup_)
{
HashMap<String, Object> setup_vals = (HashMap<String, Object>)arrays.get_new(dbs_setup_);
String db = common.DEFAULT_DB;
String name = (String)arrays.get_value(setup_vals, accessory._types.CONFIG_DB_NAME);
if (!strings.is_ok(name)) name = common.DEFAULT_DB_NAME;
HashMap<String, Object[]> sources = new HashMap<String, Object[]>();
sources = add_source_market(db, sources);
sources = add_source_execs(db, sources);
sources = add_source_basic(db, sources);
sources = add_source_remote(db, sources);
sources = add_source_orders(db, sources);
sources = add_source_trades(db, sources);
sources = add_source_watchlist(db, sources);
sources = add_source_apps(db, sources);
boolean is_ok = populate_db(db, name, sources, setup_vals);
return is_ok;
}
private HashMap<String, Object[]> add_source_market(String db_, HashMap<String, Object[]> sources_)
{
String source = market.SOURCE;
String table = "ib_market";
HashMap<String, db_field> info = new HashMap<String, db_field>();
info.put(market.SYMBOL, common.get_field_symbol(true));
info.put(market.TIME, common.get_field_time());
info.put(market.HALTED, common.get_field_halted());
info.put(market.HALTED_TOT, common.get_field_halted_tot());
info.put(market.ENABLED, db_common.get_field_boolean(true));
info.put(market.PRICE, common.get_field_price());
info.put(market.OPEN, common.get_field_price());
info.put(market.CLOSE, common.get_field_price());
info.put(market.LOW, common.get_field_price());
info.put(market.HIGH, common.get_field_price());
info.put(market.ASK, common.get_field_price());
info.put(market.BID, common.get_field_price());
info.put(market.SIZE, common.get_field_size_volume());
info.put(market.BID_SIZE, common.get_field_size_volume());
info.put(market.ASK_SIZE, common.get_field_size_volume());
info.put(market.VOLUME, common.get_field_size_volume());
return add_source_common(db_, source, table, info, sources_);
}