DataClover

dataclover

templating is a powerful feature provided by the language that allows you to generate text output by replacing placeholders (variables) in a template with their corresponding values. It's a convenient way to generate dynamic content, including files. Go YAML , a package manager for , utilizes to generate from templates. Helm Kubernetes Go templating Kubernetes manifest files I needed the same in a project of mine. In this tutorial, we'll see how we can use Go templating to replace values in a with values from another , similar to what Helm does. YAML file YAML file The reference Github repo can be found . here Template file and values Suppose we want to replace our template with some values like Helm does. template/template.yaml apiVersion: v1
kind: Deployment
metadata:
  name: {{ .AppName }}
spec:
  replicas: {{ .ReplicaCount }}
  template:
    spec:
      containers:
      - name: {{ .AppName }}
        image: {{ .Image }} template/values.yaml AppName: my-app
ReplicaCount: 3
Image: myregistry/my-app:v1.0.0 Template parsing parser/parser.go // Copyright (c) 2023 Tiago Melo. All rights reserved.
// Use of this source code is governed by the MIT License that can be found in
// the LICENSE file.
package parser


import (
    "html/template"
    "io"
    "os"


    "github.com/pkg/errors"
    "gopkg.in/yaml.v2"
)


// For ease of unit testing.
var (
    parseFile           = template.ParseFiles
    openFile            = os.Open
    createFile          = os.Create
    ioReadAll           = io.ReadAll
    yamlUnmarshal       = yaml.Unmarshal
    executeTemplateFile = func(templateFile *template.Template, wr io.Writer, data any) error {
        return templateFile.Execute(wr, data)
    }
)


// valuesFromYamlFile extracts values from yaml file.
func valuesFromYamlFile(dataFile string) (map[string]interface{}, error) {
    data, err := openFile(dataFile)
    if err != nil {
        return nil, errors.Wrap(err, "opening data file")
    }
    defer data.Close()
    s, err := ioReadAll(data)
    if err != nil {
        return nil, errors.Wrap(err, "reading data file")
    }
    var values map[string]interface{}
    err = yamlUnmarshal(s, &values)
    if err != nil {
        return nil, errors.Wrap(err, "unmarshalling yaml file")
    }
    return values, nil
}


// Parse replaces values present in the template file
// with values defined in the data file, saving the result
// as an output file.
func Parse(templateFile, dataFile, outputFile string) error {
    tmpl, err := parseFile(templateFile)
    if err != nil {
        return errors.Wrap(err, "parsing template file")
    }
    values, err := valuesFromYamlFile(dataFile)
    if err != nil {
        return err
    }
    output, err := createFile(outputFile)
    if err != nil {
        return errors.Wrap(err, "creating output file")
    }
    defer output.Close()
    err = executeTemplateFile(tmpl, output, values)
    if err != nil {
        return errors.Wrap(err, "executing template file")
    }
    return nil
} First, we call 'template.ParseFiles' to create a new template and parse the template definitions from the named files. Then, we need to read the YAML file and parse it - our 'valuesFromYamlFile' returns a 'map[string]interface{}' containing keys and values found. We're using gopkg.in/yaml.v2 for that. Next, we create the output file in which the output will be written - that is, the template file 'template/template.yaml' with all placeholders replaced by the values we defined in 'template/values.yaml'. If you do not want to save it to a new file, you can do just 'templateFile.Execute(os.Stdout, values)' and then the output will be printed to the console. Finally, we execute the template to replace all placeholders in the template file and write the output to a new file. Using it cmd/main.go // Copyright (c) 2023 Tiago Melo. All rights reserved.
// Use of this source code is governed by the MIT License that can be found in
// the LICENSE file.
package main


import (
    "fmt"
    "os"


    "tiago.com/parser"
)


func run() error {
    const templateFile = "template/template.yaml"
    const dataFile = "template/values.yaml"
    const outputFile = "parsed/parsed.yaml"
    if err := parser.Parse(templateFile, dataFile, outputFile); err != nil {
        return err
    }
    fmt.Printf("file %s was generated.\n", outputFile)
    return nil
}


func main() {
    if err := run(); err != nil {
        fmt.Println(err)
        os.Exit(1)
    }
} Running it $ make run

file parsed/parsed.yaml was generated. Output parsed/parsed.yaml apiVersion: v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: my-app
        image: myregistry/my-app:v1.0.0 And that’s it!

Golang Templating: Substituting Values Between Different YAML Files

Lets Graph Simple Moving Averages Using Rust

How to (Re)Make A Geologic Map In Python With PyGMT

Repository: entronad/crypto-es npm: crypto-es Needs of encryption, decryption and hash are quite common in both front and back end projects. When dealing with sensitive data, functions with the name like , or may be quite often seen in your codes. MD5 Base64 AES Of all JavaScript cryptography libraries, is mostly widely used for its simple APIs and rich variety of functions. It derives in early years, and the main codebase is still on the Google Code. Although it has been migrated to npm, the last update was 3 years ago. For historical reasons, some of it’s features seem out of date for modern usages: CryptoJS It creates an unique object oriented system based on prototypal inheritance to simulate class inheritance. The files use Immediately Invoked Function Expression (IIFE) to export modules. Before ECMAScript 6, these features avoided certain defects of JavaScript while ensuring browser compatibility and out-of-the-box usage. But since the new ECMAScript standard has specified the Class and Module system to solve those defects, I would try to experimentally refactor CryptoJS in modern ECMAScript. The project is called . As an experiment, the compatibility would not be the first concern, and the only rule is to meet the latest ECMAScript standard. For example, I would only use ECMAScript Module for the module system, with out CommonJS support. With the help of Babel and loader hook, this project could satisfy any production application. And as the ECMAScript standard gets more widespread, there would be more directly usages of it. CryptoES Besides, as a refactoring project, it’s APIs is all the same as CryptoJS. Refactoring to the ECMAScript Class CryptoJS extends the JavaScript prototypal inheritance and implements an unique “base objects inheritance” system. It has functions such as extend, override, mixin, etc. which makes it like general class-based OOP languages. This is defferent in approach but the same in purpose of the ECMAScript Class. Our first step is to replace this unique inheritance system with classes, which would make the code concise. But before that, let's have a review of some key concepts of the ECMAScript Class: 1. constructor We know that in ECMAScript, class definition is a syntactic sugar of traditional JavaScript constructor. So we could get access to an instance’s class through it’s constructor field. What could we do with this knowledge? In an instance's instance methods, we could create a new instance which has the same class as the former one by expression new . this.constructor() 2. this Unlike in the traditional JavaScript prototypal inheritance, in ECMAScript Class inheritance, while creating an instance, the constructor of it’s super class will firstly called, and then the fields and methods of the super class will be added to the instance, finally the constructor of the instance will be called. This makes sure that the fields and methods defined in the sub class will override the super class’s correctly. Note that this in instance methods refers to the instance, while this in static methods refers to the class. So we could implement factory mode by calling new this() in the static methods of a class. 3. super In the class definition, with parentheses refers to super class's constructor, while super without parentheses in static methods, just like this, refers to super class. So, when overriding methods, we could call the overridden method of the super class first by expression . super() super.overridedMethod.call(this) 4. prototype & __proto__ A class is a constructor in essence, so it has a prototype field referring to its prototype object. And a class is also an object, so it has a field referring to it's super class too. Since the prototype of a class has a field referring to it's super class's prototype object, this composes a prototype chain. __proto__ __proto__ An instance’s field refers to the prototype object of it's class. __proto__ With these features, we could get the inheritance relations of both instances and classes. The commonly used version of CryptoJS nowadays is hosted on the GitHub: ，which will mainly based on. It’s “base objects” are defined in the file. brix/crypto-js CryptoES core.js The inheritance is implemented in the object named Base: Base = C_lib.Base = ( { { : { subtype = create( );
​ (overrides) {
        subtype.mixIn(overrides);
      }
​ (!subtype.hasOwnProperty( ) || .init === subtype.init) {
        subtype.init = {
        subtype.$ .init.apply( , );
      };
    }
​ subtype.init.prototype = subtype;
​ subtype.$ = ;
​ subtype;
    },
​
    create: { instance = .extend();
      instance.init.apply(instance, );
​ instance;
    },
​
    init: {
    },
  };
}()); var ( ) function return extend ( ) function overrides // Spawn var this // Augment if // Create default initializer if 'init' this ( ) function super this arguments // Initializer's prototype is the subtype object // Reference supertype super this return ( ) function var this arguments return ( ) function In fact, the inheritance is implemented by calling the extend method of the "super object" with arguments of overriding fields and methods, and the extend method creates a new "sub object". The real instance will be returned by the create method of this "sub object", of which the init method will be called as a constructor. The disadvantage of this approach is that instances are not created with the keyword new as regular. And an instance will recursively keeps all the "ancestor objects" of its inheritance chain in the $super field, which is certainly redundant. These goals could easily achieved with the keyword extend and class constructor of ECMAScript Class, with out any extra codes or problems above. But for the consistence of the APIs, the static method create is kept, so that you could create instances with . The arguments of the create method will be passed to the real constructor by the rest operator and the argument destruction: ClassName.create() { create(...args) { (...args);
  }
} export class Base static return new this the basic purpose of Base is to provide the mixin method. In CryptoJS, the boundary between "class objects" like Base and real instance objects is fuzzy, and the mixin method could be called by "class objects": mixIn: { ( propertyName properties) { (properties.hasOwnProperty(propertyName)) { [propertyName] = properties[propertyName];
    }
  }
​ (properties.hasOwnProperty( )) { .toString = properties.toString;
  }
}, ( ) function properties for var in if this // IE won't copy toString using the loop above if 'toString' this But in both logic and fact, the mixin method should be an instance method, which just acts like : Object.assign() mixIn(properties) { .assign( , properties);
} return Object this Finally, the copying of instances. With prototypal inheritance, the CryptoJS implementation is quite odd: clone() { clone = .constructor(); .assign(clone, ); clone;
} const new this Object this return Since new could be used in any instances with out indicating it's the class name, we could do it in a more straight way: this.constructor() clone() { clone = .constructor(); .assign(clone, ); clone;
} const new this Object this return Then, by inheriting the Base class, all other core classes will have these methods. And with ECMAScript Class, the usage of methods would be more standardized. For example, in CryptoJS, some instance creating methods of Wordarray are: WordArray = C_lib.WordArray = Base.extend({ : {
    words = .words = words || [];
​ (sigBytes != ) { .sigBytes = sigBytes;
    } { .sigBytes = words.length * ;
    }
  }, : {
    ...
    return WordArray.init(words, nBytes);
  }
});

But now they are standard or static methods:

export class WordArray extends Base { (words = [], sigBytes = words.length * 4) { ();
​ .words = words; .sigBytes = sigBytes;
  }
​ random(nBytes) {
    ...
    return WordArray(words, nBytes);
  }
} var init ( ) function words, sigBytes this if undefined this else this 4 random ( ) function nBytes new constructor constructor super this this static new After this refactoring to class, we should add some extra unit test for the inheritance relations: of the sub class must refer to the super objects of sub class and super class are of right order in the prototype chain. __Proto__ class.prototype prototype objects of sub class and super class are of right order in the prototype chain. data.Obj = {
};

data.obj = data.Obj.create();

it( , () => {
  expect(data.Obj.__proto__).toBe(C.lib.Base);
});

it( , () => {
  expect(data.obj.__proto__.__proto__).toBe(C.lib.Base.prototype);
}); . . class Obj extends C lib Base 'class inheritance' 'object inheritance' WordArray and Bitwise Operations Bitwise operations are the foundation of cryptography algorithms. What ever the data type is, bitwise operation treats them as a contiguous sequence. For sack of performance, bitwise operation would better acting on a section of contiguous memory. Some languages provide ways to operate the contiguous memory, such as pointer in C++ and ArrayBuffer in ECMAScript 6. JavaScript was designed as a script language for browser at first, so there was no memory operation before. But there is still a way to get the abstraction of the bitwise sequence, which is Binary Bitwise Operators. According to the specification, in the operation with binary bitwise operators, all operands, whatever their origin types are, will be convert to 32 bits unsighed int by ToInt32(). Then they are treated as 32 bits length sequences, and the result is a 32 bits unsighed int. So, by splicing these 32 bits unsighed ints, we could simulate the operation on a contiguous memory. Based on this, CryptoJS implemented a class called WordArray, as the abstraction of the bitwise sequence. WordArray is the most important basic class of CryptoJS, and all it’s algorithms handle with WordArray objects in underlying implementation. Thus understanding WordArray is the precondition to understand it’s algorithms. The definition of WordArray is in the file: core.js Note that all codes below are of . entronad/crypto-es { (words = [], sigBytes = words.length * 4) { (); .words = words; .sigBytes = sigBytes;
  }
  
  ...
} export class WordArray extends Base constructor super this this WordArray directly inherits from Base . It has two fields: word and sigBytes. words is an array of 32 bits unsighed ints, and By splicing elements of this array in order we can get the bitwise sequence needed. In JavaScript conversion between the 32 bits unsighed int and the bit is through Binary Complement. But we don't have to get involved with that because the value of this int is meaningless. Generally the bitwise sequence is measured by bytes, or presented as hex numbers, so we just need to know that 32 bits equal to 4 bytes, or 8 hex numbers. Encoding algorithms’ subjects are strings. So the bitwise sequences are all of whole bytes, or times of 8 bits. But they are not certainly times of 32 bits. So only by the length of words we can't get the actual length of bitwise sequence, for there may be some empty tailing bytes. So we need the field sigBytes indicates the actual significant bytes length. We could create a WordArray by directly passing these two fields: wordArray = CryptoES.lib.WordArray.create([ , ], ); const 0x00010203 0x04050607 6 For convenience of pruning the words with the sigBytes, there is a clamp method in WordArray: clamp() { { words, sigBytes } = ;
​ words[sigBytes >>> ] &= << ( - (sigBytes % ) * );
  words.length = .ceil(sigBytes / );
} // Shortcuts const this // Clamp 2 0xffffffff 32 4 8 Math 4 It will remove the “insignificant bytes” of the words . In the words array, the starting elements full of significant bytes will be kept, and the tailing elements with no significant bytes will be ignored through . words.length = Math.ceil(sigBytes / 4) The middle element with both significant and insignificant bytes is a bit difficult to deal with. Firstly we should calculate the length to remove: , and left move bitwise by this length to get a 32 bits mask, then locate this middle element by (just the same as int divided by 4) , finally and it with the mask to set the insignificant bytes to 0. (32 - (sigBytes % 4) * 8) 0xffffffff sigBytes >>> 2 Locating elements by and making and with masks is widely used in CryptoJS. >>> Just like clamp, the troublesome part of concat is also to handle the middle element: concat(wordArray) { thisWords = .words; thatWords = wordArray.words; thisSigBytes = .sigBytes; thatSigBytes = wordArray.sigBytes;
​ .clamp();
​ (thisSigBytes % ) { ( i = ; i < thatSigBytes; i += ) { thatByte = (thatWords[i >>> ] >>> ( - (i % ) * )) & ;
      thisWords[(thisSigBytes + i) >>> ] |= thatByte << ( - ((thisSigBytes + i) % ) * );
    }
  } { ( i = ; i < thatSigBytes; i += ) {
      thisWords[(thisSigBytes + i) >>> ] = thatWords[i >>> ];
    }
  } .sigBytes += thatSigBytes;
​ ;
} // Shortcuts const this const const this const // Clamp excess bits this // Concat if 4 // Copy one byte at a time for let 0 1 const 2 24 4 8 0xff 2 24 4 8 else // Copy one word at a time for let 0 4 2 2 this // Chainable return this Inside CryptoJs, WordArray is both input and output of most functions, but the external users concerns mostly about the string result. So WordArray provides overriding toString method: toString(encoder = Hex) { encoder.stringify( );
} return this Because the words array is of reference type, we should create a new copy of it by slice in the clone method: clone() { clone = .clone.call( );
  clone._data = ._data.clone();
​ clone;
} const super this this return Except the constructor, the static method random provides a random WordArray of certain length. Because of JavaScript is unsafe and returns a 64 bits float, we will do some extra processing: Math.random() random(nBytes) { words = [];
​ r = { _m_w = m_w; _m_z = ; mask = ;
​ {
      _m_z = ( * (_m_z & ) + (_m_z >> )) & mask;
      _m_w = ( * (_m_w & ) + (_m_w >> )) & mask; result = ((_m_z << ) + _m_w) & mask;
      result /= ;
      result += ; result * ( .random() > ? : );
    };
  };
​ ( i = , rcache; i < nBytes; i += ) { _r = r((rcache || .random()) * );
​
    rcache = _r() * ;
    words.push((_r() * ) | );
  }
​ WordArray(words, nBytes);
} static const const ( ) => m_w let let 0x3ade68b1 const 0xffffffff return => () 0x9069 0xFFFF 0x10 0x4650 0xFFFF 0x10 let 0x10 0x100000000 0.5 return Math 0.5 1 -1 for let 0 4 const Math 0x100000000 0x3ade67b7 0x100000000 0 return new ArrayBuffer and TypedArray Since the ArrayBuffer is included to the ECMAScript, it has been used in more and more scenes, such as WebSocket, file objects, and canvas outputs. While handling with these forms of objects, we also need to encrypt, decrypt or hash them in some cases. So CryptoJS extended the WordArray creator to allow ArrayBuffer and TypedArray as input. this extension is in a individual file, and dose a lot of checks and reconstructs the WordArray creator to ensure the compatibility. We integrate this to the origin WordArray constructor and simplified the these checks: lib-typedArrays.js (words = [], sigBytes = words.length * 4) { ();
​ typedArray = words; (typedArray ) {
    typedArray = (typedArray);
  }
​ (
    typedArray || typedArray || typedArray || typedArray || typedArray || typedArray || typedArray || typedArray ) {
    typedArray = (typedArray.buffer, typedArray.byteOffset, typedArray.byteLength);
  }
​ (typedArray ) { typedArrayByteLength = typedArray.byteLength;
​ _words = []; ( i = ; i < typedArrayByteLength; i += ) {
      _words[i >>> ] |= typedArray[i] << ( - (i % ) * );
    }
​ .words = _words; .sigBytes = typedArrayByteLength;
  } { .words = words; .sigBytes = sigBytes;
  }
} constructor super let // Convert buffers to uint8 if instanceof ArrayBuffer new Uint8Array // Convert other array views to uint8 if instanceof Int8Array instanceof Uint8ClampedArray instanceof Int16Array instanceof Uint16Array instanceof Int32Array instanceof Uint32Array instanceof Float32Array instanceof Float64Array new Uint8Array // Handle Uint8Array if instanceof Uint8Array // Shortcut const // Extract bytes const for let 0 1 2 24 4 8 // Initialize this word array this this else // Else call normal init this this Then WordArray creator could recive an ArrayBuffer or TypedArray so that CryptoES algorisms could apply to them: Then WordArray creator could recive an ArrayBuffer or TypedArray so that CryptoES algorisms could apply to them: words = CryptoES.lib.WordArray.create( ( )); rst = CryptoES.AES.encrypt(words, ) const new ArrayBuffer 8 const 'Secret Passphrase' Note that ArrayBuffer could not directly passed to algorisms, you should change them to WordArray first. With this, encrypting files would be easier fileInput = .getElementById( ); file = fileInput.files[ ]; reader = FileReader();
reader.readAsArrayBuffer(file);
reader.onload = { arrayBuffer = reader.result; words = CryptoES.lib.WordArray.create(arrayBuffer); rst = CryptoES.AES.encrypt(words, )
  ...
}; const document 'fileInput' const 0 const new ( ) function const const const 'Secret Passphrase' Stay Tuned! Previous published at https://medium.com/front-end-weekly/refactoring-cryptojs-in-modern-ecmascript-1d4e1837c272

Refactoring CryptoJS in Modern ECMAScript

Why you should be cautious with neural networks for trading So I built a Deep Neural Network to predict the price of Bitcoin — and it’s astonishingly accurate. Curious? See the prediction results for yourself. Looks pretty accurate, doesn’t it? And before you ask: Yes, the above evaluation was performed on unseen test data — only prior data was used to train the model (more details later). So this is a money-making machine I can use to get rich! Right? In fact, I am giving you the for the above model so that you can use it yourself… code Ok, . Don’t do it. stop right there I repeat: Do not use it for trading. Don’t do it! Don’t be fooled. There is something utterly deceptive about these results. Let me explain. Too Good to be True During the last couple of weeks and months I’ve encountered many articles that take a similar approach to the one presented here and that show graphs of cryptocurrency price predictions that look like the one above. The seemingly stunning accuracy of price predictions should immediately set off alarm bells. These results are obviously too good to be true. “When something looks too good to be true, it usually is.” — Emmy Rossum In the following, I want to demonstrate why this is the case. Don’t get me wrong — my intention is not to undermine the work put into those articles. They are good and deserve the claps they received. In fact, many of those approaches are very accurate — . technically speaking The goal of this article is to bring out why those models are, fallacious and why their predictions are not necessarily suitable for usage in actual trading. in practice, So why exactly is this the case? Let’s take a close look. Predicting the Price of Bitcoin using LSTMs To explain, let me walk you through an example of building a multidimensional neural network to predict the price of Bitcoin that yields the prediction results you saw above. Long Short Term Memory (LSTM) LSTMs are a special kind of , that are particularly suitable for time series problems. Hence, they have become popular when trying to forecast cryptocurrency prices, as well as stock markets. Recurrent Neural Networks (RNN) For in-depth introductions to LSTMs I recommend and article. this this For the present implementation of the LSTM, I used Python and . Keras (You can find the corresponding Jupyter Notebook with the complete code on my Github .) 1. Getting the Data First, I fetched historic Bitcoin price data (you can do this for any other cryptocurrency as well). To do so I used the API from : cryptocompare import jsonimport requestsimport pandas as pd endpoint = 'https://min-api.cryptocompare.com/data/histoday'res = requests.get(endpoint + '?fsym=BTC&tsym=USD&limit=2000')hist = pd.DataFrame(json.loads(res.content)['Data'])hist = hist.set_index('time')hist.index = pd.to_datetime(hist.index, unit='s')hist.head() A snapshot of historic Bitcoin price data. Voilà, historic daily BTC data for the last days, from until . 2000 2012–10–10 2018–04–04 2. Train-Test Split Then, I split the data into a and a set. I used the last of the data for testing, which splits the data on the All data before this date was used for training, all data from this date on was used to test the trained model. Below, I plotted the column of our DataFrame, which is the daily closing price I intended to predict. training test 10% 2017–09–14. close def train_test_split(df, test_size=0.1):split_row = len(df) - int(test_size * len(df))train_data = df.iloc[:split_row]test_data = df.iloc[split_row:]return train_data, test_data def line_plot(line1, line2, label1=None, label2=None, title=''):fig, ax = plt.subplots(1, figsize=(16, 9))ax.plot(line1, label=label1, linewidth=2)ax.plot(line2, label=label2, linewidth=2)ax.set_ylabel('price [USD]', fontsize=14)ax.set_title(title, fontsize=18)ax.legend(loc='best', fontsize=18) train, test = train_test_split(hist, test_size=0.1)line_plot(train.close, test.close, 'training', 'test', 'BTC') Train-test split of historic Bitcoin price data 3. Building the Model For training the LSTM, the data was split into windows of days (this number is arbitrary, I simply chose a week here) and within each window I normalised the data to , i.e. the first entry of each window is and all other values represent the change with respect to the first value. Hence, I am predicting price , rather than absolute price. 7 zero base 0 changes def normalise_zero_base(df):""" Normalise dataframe column-wise to reflect changes withrespect to first entry."""return df / df.iloc[0] - 1 def extract_window_data(df, window=7, zero_base=True):""" Convert dataframe to overlapping sequences/windows oflength `window`."""window_data = []for idx in range(len(df) - window):tmp = df[idx: (idx + window)].copy()if zero_base:tmp = normalise_zero_base(tmp)window_data.append(tmp.values)return np.array(window_data) def prepare_data(df, window=7, zero_base=True, test_size=0.1):""" Prepare data for LSTM. """# train test splittrain_data, test_data = train_test_split(df, test_size) # extract window data  
X\_train = extract\_window\_data(train\_data, window, zero\_base)  
X\_test = extract\_window\_data(test\_data, window, zero\_base)  
  
# extract targets  
y\_train = train\_data.close\[window:\].values  
y\_test = test\_data.close\[window:\].values  
if zero\_base:  
    y\_train = y\_train / train\_data.close\[:-window\].values - 1  
    y\_test = y\_test / test\_data.close\[:-window\].values - 1

return train\_data, test\_data, X\_train, X\_test, y\_train, y\_test train, test, X_train, X_test, y_train, y_test = prepare_data(hist) I used a simple neural network with a single LSTM layer consisting of neurons, a dropout factor of , and a Dense layer with a single activation function. In addition, I used as loss function and the optimiser. 20 0.25 linear Mean Absolute Error (MAE) Adam I trained the network for epochs with a batch size of . 50 4 Note: The choice of the network architecture and all parameters is arbitrary and I didn’t optimise for any them, as this is not the focus of this article. def build_lstm_model(input_data, output_size, neurons=20,activ_func='linear', dropout=0.25,loss='mae', optimizer='adam'):model = Sequential() model.add(LSTM(neurons, input\_shape=(  
          input\_data.shape\[1\], input\_data.shape\[2\])))  
model.add(Dropout(dropout))  
model.add(Dense(units=output\_size))  
model.add(Activation(activ\_func))

model.compile(loss=loss, optimizer=optimizer)  
return model model = build_lstm_model(X_train, output_size=1)history = model.fit(X_train, y_train, epochs=50, batch_size=4) 4. Results Using the trained model to predict on the left-out test set, we obtain the graph shown in the beginning of this article. So what exactly is wrong with these results? Why shouldn’t we use this model for actual trading? Let’s take a closer look and zoom into the last days of the plot. 30 targets = test[target_col][window:]preds = model.predict(X_test).squeeze()# convert change predictions back to actual pricepreds = test.close.values[:-window] * (preds + 1)preds = pd.Series(index=targets.index, data=preds) n = 30 line_plot(targets[-n:], preds[-n:], 'actual', 'prediction') See that? You might have already correctly guessed that the fundamental flaw with this model is that for the prediction of a particular day, it is mostly using the value of the previous day. The prediction line doesn’t seem to be much more than a shifted version of the actual price. In fact, if we adjust the predictions and shift them by a day, this observation becomes even more obvious. line_plot(targets[-n:][:-1], preds[-n:].shift(-1)) As you can see, we suddenly observe an almost perfect match between actual data and predictions, indicating that the model is essentially the price at the previous day. learning These results are exactly what I’ve been seeing in many of the examples using single-point predictions with LSTMs. To make this point clearer, let’s compute the expected as predicted by the model and compare those with the actual returns. returns actual_returns = targets.pct_change()[1:]predicted_returns = preds.pct_change()[1:] Looking at the actual and predicted returns, both in their original form as well as with the applied to them, we obtain the same observation. 1-day-shift Actual and predicted returns. In the left plot predictions are adjusted by a day. Actually, if we compute the correlation between actual and predicted returns both for the original predictions as well as for those adjusted by a day, we can make the following observation: fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(18, 9)) # actual correlationcorr = np.corrcoef(actual_returns, predicted_returns)[0][1]ax1.scatter(actual_returns, predicted_returns, color='k')ax1.set_title('r = {:.2f}'.format(corr), fontsize=18) # shifted correlationshifted_actual = actual_returns[:-1]shifted_predicted = predicted_returns.shift(-1).dropna()corr = np.corrcoef(shifted_actual, shifted_predicted)[0][1]ax2.scatter(shifted_actual, shifted_predicted, color='k')ax2.set_title('r = {:.2f}'.format(corr)); As you can see from the plots above, actual and predicted returns are uncorrelated. Only after applying the on the predictions we obtain highly correlated returns that resemble the returns of the actual bitcoin data. 1-day-shift Summary The goal of the this blogpost was to address the many examples of predictions of cryptocurrency and stock market prices using deep neural networks that I have encountered in the past couple of months — these take a similar approach as the one employed here: Implementing an LSTM using historic price data to predict outcomes. I have demonstrated why these models might not be necessarily viable for actual trading. future Yes, the network is effectively able to learn. But it ends up using a strategy in which predicting a value close to the previous one turns out to be successful in terms of minimising the mean absolute error. However, no matter how accurate the predictions are in terms of the loss error — in practice, the results of single-point prediction models based on , as the one showcased here, remain hard to accomplish and are not particularly useful for trading. historic price data alone Needless to say that more sophisticated approaches of implementing useful LSTMs for price predictions potentially do exist. Using more data, as well as optimising network architecture and hyperparameters are a start. In my opinion, however, there is more potential in incorporating data and features that go beyond historic prices alone. After all, the finance world has already known for long that “ ”. past performance is not an indicator for future outcomes And the same might also hold for cryptocurrencies. Disclaimer: This is not financial advice. The article and the presented model are for educational purposes only. Do not use it for trading or making investment decisions.

Don’t be fooled — Deceptive Cryptocurrency Price Predictions Using Deep Learning

Professor Robert J. Stone, who leads the VR/Human Interface Technologies team at the University of Birmingham, celebrates his 30-year anniversary in the field of Virtual Reality this year (2017). Among his VR ‘hits’ that he had developed include: one of the first & Simulation (VRS) initiative in 1993, invention of one of the first (if not first) tactile feedback gloves back in 1992, and the design and development of the world’s first keyhole surgery skills trainer, MIST_VR_ in 1995. MIST_VR_ became the world’s first part-task surgical skills trainer, adopted and evaluated on an international scale, and adopted as the skills trainer by the European Surgical Institute in Germany. Virtual Reality the de facto Vanessa Radd caught up with Stone, who is also a founding member of the XR Alliance. Vanessa Radd: Bob, first of all, congratulations on your 30th anniversary in VR! Thank you so much for sharing your experience with us and thoughts on how you see the future of VR (and AR). I believe all of us have a lot to learn from your experience especially in implementing human-centered design and getting VR projects out of the lab to real life deployments. Could you start by sharing with us about your background? Bob Stone: My academic background is in Applied Psychology and Human Factors (a.k.a. Ergonomics). In brief, upon graduating from UCL, I started my career as a Human Factors specialist at what was British Aerospace in Bristol, UK, undertaking work in the defence, subsea oil and gas, and space domains, and then, following an amazing experience at NASA’s Research Centre in California in 1987, I joined the UK’s National Advanced Robotics Research Centre in 1979 to undertake research and development in the field of VR and “telepresence”. Then, following the successful launch and completion of a totally industry-funded Virtual Reality & Simulation initiative between 1993 and 1995, I became the Director of a small spin-out company called VR Solutions. This company went through many roller coaster transitions over a 7-year period, including being purchased by two US organizations, and I finally decided to make the “jump” to academia in 2003. Today, I lead a small group of researchers and students within the School of Engineering at the University of Birmingham called the Human Interface Technologies (HIT) team. Vanessa Radd: Please tell us about the work you do. In what industry verticals are you and your team focusing on? Bob Stone: As for industry sectors, my team and I tend to get involved wherever we think there is a particular technical challenge or where we can help — from an independent perspective — potential adopters to make the right choices of VR or AR for their organizations and end users. However, in terms of the sectors we “specialize” in, based on where the main — and sustainable — projects are evolving, I would say we are particularly active in defense, healthcare and heritage. technology In the defense area, for example, we have a particularly strong track record in areas such as simulation for submarine, counter-terrorist and bomb disposal robot training, Mixed Reality, or “MxR” techniques for command and control, and, last but not least, defense healthcare — MxR training for future Armed Forces paramedics, for example. In healthcare generally, we have, in the past, developed VR and AR for surgical training and education; today, we’re investigating the delivery of virtual scenes of nature into hospital intensive care wards to help improve patient sleep quality and post-operative rehabilitation. Finally, in the heritage sector, which is my personal “favorite” (because it enables us to work closely with schoolchildren and members of the public), we undertake projects as wide-ranging as using drones to recreate VR models of remote areas of historical significance, subsea wreck visualizations, and — in collaboration with colleagues in the US — the development of VR and AR content to support the educational aspects of the commemorations scheduled for 2020. Mayflower 400 Vanessa Radd: How much of your work delve into VR vs AR? Bob Stone: Historically, my work and that of my team has focused on VR, but today, especially with many of our project areas focusing on defence, healthcare and heritage, we’re probably expanding as much effort into AR and MxR as we are in VR. I predict a strong growth in MxR over the coming years, as I believe that blending the “best of the real” with the “best of the virtual, or augmented” will deliver the most credible and affordable solutions in training and real-time visualization. Having said that, I have to confess that my personal technical implementation skills when it comes to VR, AR or MxR are quite limited! I have a small team of very competent specialists providing expertise in topics like 3D modelling, physics-based rendering and VR toolkit exploitation. My main area of expertise is, and always has been, human factors and human-centred design — trying to ensure that we deliver appropriate interactive hardware technologies and appropriate content to real people undertaking real tasks in the real world. For VR, AR and MxR to succeed, especially given the academic nature of my team, we cannot simply develop our solutions and demonstrators in a cosy academic laboratory, publishing papers in journals that those who might benefit from our research simply will not read!! We MUST work with the end users from the beginning, and throughout each and every stage of the project. Vanessa Radd: Tell us more about what you have developed in VR and AR. Bob Stone: Over the years, both my commercial and academic teams have developed a huge range of what we call “concept capability demonstrators”. What this means is that we take on small, self-contained projects over short periods of time, working closely with sponsors to develop something they and their colleagues or employees can evaluate, prior to making significant and highly risk-laden investments in otherwise unproven technologies (we’re seeing a lot of that in the VR and AR community today, sadly). My team’s portfolio is huge, with far too many projects to summarize here. But they are very diverse indeed, ranging from submarine safety training to future AR and MxR concepts for military/security service command and control and next-generation aircraft concepts; from recreating historical wrecks and subsea habitats in Plymouth Sound to abandoned railway lines and archaeological sites on Dartmoor; and from future autonomous vessels to novel ideas for AR exhibits with the UK’s National Marine Aquarium. Very occasionally we are fortunate to be in a position to take our developments to market, or to the end users for trials and evaluation at the very least. At the end of the 1990s, a close-range gunnery VR training system, developed by my previous commercial VR team, was installed at the Royal Navy’s land base at HMS Collingwood, and was used for basic weapons training for many years. Similarly, helicopter rear-crew VR systems were delivered to, and regularly used for training by the RAF at their search-and-rescue bases in Shawbury (England) and Valley (in Wales). An avionics maintenance training system, based on an extensive VR model of an RAF aircraft was also installed at a British RAF base and used very effectively for many years. One particular success in respect of taking solutions to “market” is the remote driving and manipulation simulator for the UK’s current bomb disposal robot, . After some 3 to 4 years of development and testing using VR software toolkits, we (uniquely for a British University) were sponsored to join forces with a small local company to deliver 42 simulator units to Explosive Ordnance Disposal (EOD) squadrons across the whole of the UK, and to Cyprus and Gibraltar as well. Tornado CUTLASS Vanessa Radd: What was your biggest achievement in VR to date? What are you most proud of? Bob Stone: Difficult question, as practically every month has brought with it a new challenge, enabling us to test some of today’s VR and AR technologies to their limits. We don’t always succeed, of course, but when we do, it’s always a huge and exciting achievement for the team! Looking back over the past 30 years, there are a number of achievements I am very proud of. Launching the Virtual Reality & Simulation, or initiative, at VR Solutions Ltd back in 1993, when we were still part of the UK’s National Advanced Robotics Research Centre. The response we had from industry, following a chance appearance of my team on the BBC’s in January 1993, was tremendous. So much so, that we were able to pull together companies large and small to sponsor a 3-year research and demonstrator programme without any UK Government or Research Council support at all! Just some of the names we had onboard the programme were British Nuclear Fuels, GEC Alsthom, ICI, Rolls-Royce PLC, and Vickers Shipbuilding and Engineering Limited (today part of the BAE Systems empire). VRS 9 O’Clock News VRS Another achievement, albeit rather short-lived, unfortunately, was inventing one of the first (if not first) tactile feedback gloves back in 1992. was a pneumatic prototype, initially consisting of 20 small “airpockets” distributed throughout the lower part of a Lycra glove and fed by micro-capillary tubes. It was originally designed to be used in conjunction with VPL’s . In fact, a commercial agreement along these lines was under development when, sadly, VPL filed for bankruptcy in 1990. the Teletact DataGlove I guess the one achievement I’m most proud of (as it has influenced practically every VR, AR and MxR project since) was the design and development of the world’s first keyhole surgery skills trainer, MIST_VR_ in 1995. Briefly, MIST_VR_ evolved from a project sponsored in 1994 by the UK’s Wolfson Foundation and Department of Health, the overarching aim of which was to assess the potential of emerging VR technologies to deliver cost effective keyhole surgery training for future medics. MIST_VR_ became the world’s first part-task surgical skills trainer, adopted and evaluated on an international scale, and adopted as the skills trainer by the European Surgical Institute in Germany. de facto More importantly, however, the MIST_VR_ project demonstrated conclusively that, by adopting a strong human-centered design approach from the outset of a VR project, it was all too easy to be influenced by the availability of expensive, glamorous and unproven technologies and to invest very heavily to achieve the highest fidelity (and equally expensive) solution. not Spending four afternoons in the operating theaters observing surgeons performing a variety of keyhole surgeries enabled us to design and develop a much simpler — and affordable — solution, ultimately (as the medical community confirmed) delivering exactly what the trainee surgeons required. Today, I am very proud to lead a team of very capable research scientists and VR specialists. We’re also very lucky to have attracted in the past — and, indeed, continue to attract — some incredibly keen and able students to undertake projects with us, at both undergraduate and postgraduate levels. The HIT team is very small indeed, and our funding levels are tiny, certainly in contrast to the crazy sums of money changing hands in the VR world at the moment (although we live in hope that, one day, we might find an enthusiastic benefactor who might sponsor us to do more of what we do — and work with us to take our solutions to market!). However, despite our size, we very much “punch above our weight” and people often surprised at the wide range of activities we undertake and the speed and quality with which we deliver. Vanessa Radd: What are the most successful and practical applications of VR and emerging technologies going forward — where do you see the game-changing developments? And in what industries? Bob Stone: Commentators on the VR and AR circuit are always harping on about the absence of a “killer app” for VR and AR. It’s been like that for years! In some respects, though, I would agree that there is a perception that either technology has yet to make a significant — and properly documented (i.e. not “over-glossed” with PR hype) — impact in a non-gaming sector. Indeed, VR and AR have some way to go before being mainstream in the gaming arena too. Nevertheless, since the end of VR’s second meteoric rise and fall (i.e. the early-to-mid 2000s), the affordability and accessibility of VR and, to a lesser extent (currently) AR, in terms of available hardware devices and software toolkits, has opened up a raft of opportunities in a growing number of applications. Training is an obvious example, and there are numerous case studies now becoming evident where the exposure of trainees to virtual recreations of tasks and contexts in such sectors as defense and healthcare (which includes surgical training, “training to challenge” (i.e. the decisions of one’s seniors) and various aspects of patient rehabilitation) is delivering a positive and proven advantage over and above other forms of training delivery. Other key sectors, including aerospace, automotive, maritime, to mention but three, are also prime beneficiaries for the technology, but really need to do more to demonstrate a real impact on their employees and organizations — PR comments from senior corporate personnel claiming improved time to market as a result of significant investments in VR studios are not enough. I believe that VR and AR will — indeed are — revolutionizing the education sector, at all age levels and in many, many topics, from science and history, to foreign language training and humanities. However, much more needs to be done to ensure that the confidence of teaching personnel — many of whom are still hopelessly outperformed by their students when it comes to real-time gaming and online competence — is adequately dealt with during the early stages of design! I also believe that Virtual Heritage, although not an “industry” as such, is a highly rewarding sector for applications of VR and AR technology. Although I accept (and speak from hard experience!) that heritage pursuits are never going to make significant amounts of money for anyone in the VR community, the impact the projects make, in terms of public engagement and outreach to schoolchildren, is tremendous. The demos we have staged at urban and rural school, small churches, beach clubs and for small village historical societies, not to mention the follow-up contacts such demos stimulate, constitute the most rewarding experiences I have ever had in my 30 years of involvement in VR and AR! **What are your thoughts about VR of yesteryear and the VR ‘hype’ and innovation of today?**The VR innovators of yesterday are people like Scott Fisher and Steve Ellis, Tom Furness, Fred Brooks, Myron Krueger, Morton Heilig, Bob Jacobson, Ben Delaney (editor of ), and the (sadly) late Mike Bevan (editor of the original )…that’s just a handful. CyberEdge Journal VR News Sadly, I’ve not really witnessed any one personality during the recent 3 or 4 years — in other words during the “resurrection” of VR and AR — who come anywhere close to what the VR “veterans” of yesteryear achieved. Sure, there are those animated individuals who do very well at convincing others by their animated and over-hyped on-stage appearances at the ever-increasing number of same-old, same-old VR and AR conferences. I’d like to think that my past 30 years (and the fact that I was there at the very start of VR — including the incredibly important Conference in Santa Barbara in 1990), and the fact that I’m still here (!), has, perhaps, put me on a par with these good folks, especially in terms of “flying the flag” for the UK’s role (at least) in the development of the global VR community. I just hope that the VR and AR sectors mature quickly and that we see strong examples of real benefit being reported by commercial and industrial organizations. Teleoperators & Virtual Environments We currently find ourselves in a similar situation to that experienced during the 1990s, when the hype and false promises put out by the fledgling worldwide VR “movement” resulted in many examples of premature investment in technologies and non-business-savvy start-ups that were destined to fail. Worse still, many, many potential adopters during that period had their “fingers burned” as a result of believing that the technology could deliver much more than was the truth. Hopefully we can ride the inevitable storm that approaches, so that VR and AR can, once again, progress — albeit much more slowly than most would like — to another historical period of consolidation and development. Vanessa Radd: Thank you Bob and Happy 30th VR birthday! Twitter About the Author Vanessa Radd is ranked the global top 3 augmented reality influencer and is a founding member of the XR Alliance. Vanessa is also the President of the global VR AR Association Singapore chapter. @vanradd Radd’s background comprises +17 years of internet and broadcast content & technology across the media value chain (from production, post production and content delivery, to hardware, software, image processing and cloud/virtualization – utilized as the broadcast industry standard for the Olympics) experience representing international media and technology brands in Asia. Radd also worked for a broadcast industry organization that oversaw the Olympics broadcast rights for over 3 billion viewers in Asia-Pacific and the setting of broadcast technical standards. Radd’s international executive liaison includes Tier-1 broadcasters, corporate CXOs, media organizations, non-profits, governmental agencies and the tech and developer communities. **About the XR Alliance** We at the XR Alliance work with industry partners to develop quality programs that deliver value to the ecosystem and the industry as a whole. We strongly believe in fostering good partnerships and collaborations and are selective in what we work on. We also take on suitable consulting projects, including helping VR/AR companies in the U.S. and Europe to enter Asia-Pacific markets. Please enquire about availability. E-mail: v @ xralliance.org Twitter: @xrforce

Celebrating 30 years in VR: Professor Robert J. Stone on Human Factors and the Future of VR and AR

No matter your age, background or job, you almost certainly use data daily. You might access a database to get the information necessary to run a report for your boss. Perhaps data from a mapping app helped you find your way to a new restaurant last weekend. Data also defines many interactions with professionals, from doctors to bank employees. However, a related principle called data governance underpins all those engagements. It encompasses: Who can access the data? What are the valid access reasons? How long will an organization hold the data? What does a company do with the data? Will an organization share the data with others? What are a company’s obligations for securing the data? Regardless of how your company uses data and for what reasons, it’s essential to put data governance into practice. How can you do that? 1. Identify Your Data Usage Norms Begin by clarifying your company’s relationship with data. For example, what kind of data does the organization collect and from whom? Does the content come from internal sources, external ones, or both? Is the current data volume manageable or starting to get out of control? How much do you see that changing over the next few years? The are two other things to consider. Is your data cleansed and in a format you could immediately use, or does it require preparation first? quality and sensitivity of the data Do you hold data related to someone’s finances, health, or other personal matter? If so, the information likely requires particular processing measures to remain compliant. The sooner you know how your business uses data, the easier it’ll be to establish data governance rules. However, assessing whether growth, a new product, or another change could soon alter how your company works with data is essential. If so, account for that now rather than when it might be too late. 2. Establish Access Control Measures Another vital part of data governance is ensuring people can seamlessly access the information and data-based programs necessary for their work. However, far too few companies take that approach. A 2021 Varonis study of manufacturers showed every employee to access at least 1,000 sensitive files. four in 10 businesses allowed Another astounding takeaway was that employees could access an average of six million files from their first days at work. Consider options like role-based access, taking into account that some people will rightfully access files from multiple places, especially during company-mandated travel. A related data governance strategy is to when using business data. Doing that keeps the information safer, especially when the content is intellectual property or confidential. specify an employee’s obligations 3. Create a Data Governance Team Data governance is a collective effort, so everyone is responsible for making it work. However, you’re most likely to succeed by appointing specific people to lead your business’s aims. A manager could oversee individuals tasked with particular data governance milestones. That supervisor must also clearly and its associated responsibilities. communicate a person’s data governance role Fostering an open-communication strategy is an excellent way to keep everyone motivated and on the same page. Team members should also realize data governance goals may change over time. That’s especially likely if the company begins using data differently or collecting another type. Data governance leaders should encourage people to be open to change while making it clear they’re available to hear feedback about team members’ concerns. 4. Connect Your Data Governance Practices to Company Needs Some business leaders make the mistake of thinking they should use data in specific ways, primarily because they’ve seen peers doing it. However, a better approach is to tie all data governance strategies to overarching company goals. A surprising find from a 2023 Gartner study was only believe their teams effectively provide value to their organizations. 44% of data and analytics leaders However, another statistic indicated 44% of the best-performing leaders in that group devoted time to personal development. Those who oversee data governance must determine the most practical and responsible ways to use data throughout an organization. Personal development might factor into that, such as if a data governance leader about new technologies. needs to learn more Data Governance Matters Creating and upholding a data governance plan that works for your business could take months or years — and that’s okay. The vital thing to do now is to set a firm intention of getting to that point. This is a time when consumers freely provide their data to companies for various reasons. However, they also expect the organizations to handle that information responsibly. These data governance suggestions can help you do that. Then, your business will gradually earn and maintain a reputation for handling information responsibly, which could promote growth and customer respect. Sources: https://dataconomy.com/2023/03/how-to-create-a-data-governance-strategy/?utm_content=cmp-true https://www.varonis.com/blog/manufacturing-data-risk-report-reveals-files-open-to-all-employees https://www.humanresourcesonline.net/safeguarding-a-company-s-proprietary-information-amidst-employee-exits https://www.codescan.io/blog/salesforce-data-governance-vs.-data-management/ https://www.gartner.com/en/newsroom/press-releases/03-21-2023-gartner-survey-reveals-less-than-half-of-data-and-analytics-teams-effectively-provide-value-to-the-organization https://www.analyticsinsight.net/top-10-data-governance-tools-to-know-in-the-year-2023/

Stories, News & Mentions about DATACLOVER

Dataclover’s Stories on HackerNoon