Stories by Packt on Medium

How to use Semaphores in POSIX Concurrency Control

Packt — Wed, 09 Jun 2021 11:45:15 GMT

Concept of the semaphore and its counterpart object in the pthread library: the POSIX semaphore, with an example.

Concurrency simply means having multiple pieces of logic within a program being executed simultaneously. Modern software systems are often concurrent, as programs need to run various pieces of logic at the same time. Concurrency can be implemented using only one of the multithreading or multi-processing approaches within a POSIX compliant system.

We are going to have a look at possible control mechanisms that are offered by the pthread library. Semaphores, mutexes, and condition variables alongside different types of locks are used in various combinations to bring determinism to multithreaded programs. This article is focused on Semaphores — the difference between Binary and General Semaphores and how it helps preserve Data Integrity.

This article is an excerpt from the book Extreme C by Kamran Amini. This book is a highly comprehensive resource that guides you through the most advanced capabilities of C programming.

POSIX Semaphores

In most cases, mutexes (or binary semaphores) are enough to synchronize several threads accessing a shared resource. That is because, in order to make read and write operations sequentially, only one thread should be able to enter the critical section at a time. It is known as mutual exclusion, hence, “mutex.” In some scenarios, however, you might want to have more than one thread to enter the critical section and operate on the shared resource. This is the scenario in which you should use general semaphores.

Binary semaphores

The following code is the solution made using semaphores. It involves two threads; each of them incrementing a shared integer by a different value. We want to protect the data integrity of the shared variable. Note that we won’t be using POSIX mutexes in the following code:

#include 
#include

// The POSIX standard header for using pthread library
#include

// The semaphores are not exposed through pthread.h
#include

// The main pointer addressing a semaphore object used
// to synchronize the access to the shared state.
sem_t *semaphore;

void* thread_body_1(void* arg) {
   // Obtain a pointer to the shared variable
   int* shared_var_ptr = (int*)arg;
   // Waiting for the semaphore
   sem_wait(semaphore);
   // Increment the shared variable by 1 by writing directly
   // to its memory address
  (*shared_var_ptr)++;
   printf("%d\n", *shared_var_ptr);
   // Release the semaphore
   sem_post(semaphore);
   return NULL;
}

void* thread_body_2(void* arg) {
   // Obtain a pointer to the shared variable
   int* shared_var_ptr = (int*)arg;
   // Waiting for the semaphore
   sem_wait(semaphore);
   // Increment the shared variable by 1 by writing directly
   // to its memory address
   (*shared_var_ptr) += 2;
   printf("%d\n", *shared_var_ptr);
   // Release the semaphore
   sem_post(semaphore);
   return NULL;
}
int main(int argc, char** argv) {
   // The shared variable
   int shared_var = 0;

// The thread handlers
   pthread_t thread1;
   pthread_t thread2;

#ifdef __APPLE__
   // Unnamed semaphores are not supported in OS/X. Therefore
   // we need to initialize the semaphore like a named one using
   // sem_open function.
   semaphore = sem_open("sem0", O_CREAT | O_EXCL, 0644, 1);
#else
   sem_t local_semaphore;
   semaphore = &local_semaphore;
   // Initiliaze the semaphore as a mutex (binary semaphore)
   sem_init(semaphore, 0, 1);
#endif

   // Create new threads
   int result1 = pthread_create(&thread1, NULL,
       thread_body_1, &shared_var);
   int result2 = pthread_create(&thread2, NULL,
       thread_body_2, &shared_var);
   
   if (result1 || result2) {
       printf("The threads could not be created.\n");
       exit(1);
   }
   // Wait for the threads to finish
   result1 = pthread_join(thread1, NULL);
   result2 = pthread_join(thread2, NULL);
   
   if (result1 || result2) {
      printf("The threads could not be joined.\n");
      exit(2)
   }

#ifdef __APPLE__
   sem_close(semaphore);
#else
   sem_destroy(semaphore);
#endif

   return 0;
}

The first thing you might notice in the preceding code is the different semaphore functions that we’ve used in Apple systems. In Apple operating systems (macOS, OS X, and iOS), unnamed semaphores are not supported. Therefore, we couldn’t just use sem_init and sem_destroy functions. Unnamed semaphores don’t have names (surprisingly enough) and they can only be used inside a process, by a number of threads. Named semaphores, on the other hand, are system-wide and can be seen and used by various processes in the system.

In Apple systems, the functions required for creating unnamed semaphores are marked as deprecated, and the semaphore object won’t get initialized by sem_init. So, we had to use sem_open and sem_close functions in order to define named semaphores instead.

In other POSIX-compliant operating systems, Linux specifically, we still can use unnamed semaphores and have them initialized and destroyed by using the sem_init and sem_destroy functions respectively.

In the preceding code, we have included an extra header file, semaphore.h. Inside the main function, and in Apple systems, we create a named semaphore sem0. In other POSIX-compliant operating systems, we initialize the semaphore using sem_init. Note that in this case, the pointer semaphore points to the variable local_sempahore allocated on top of the main thread’s Stack. The pointer semaphore won’t become a dangling pointer because the main thread doesn’t exit and waits for the threads to get complete by joining them.

The maximum number of threads that are allowed to be in the critical section is determined when initializing the semaphore object. We have passed the value 1 for the maximum number of threads as the last argument to the sem_open and sem_init functions; therefore, the semaphore is supposed to behave like a mutex.

To get a better understanding of semaphores, let’s dive a bit more into the details. Each semaphore object has an integer value. Whenever a thread waits for a semaphore by calling the sem_wait function if the semaphore’s value is greater than zero, then the value is decreased by 1 and the thread is allowed to enter the critical section. If the semaphore’s value is 0, the thread must wait until the semaphore’s value becomes positive again. Whenever a thread exits the critical section by calling the sem_post function, the semaphore’s value is incremented by 1. Therefore, by specifying the initial value 1, we will eventually get a binary semaphore.

General semaphores

Here we shall consider a classic example that uses general semaphores. Here the scenario in which multiple threads are allowed to enter the critical section is interesting.

This classic example involves the creation of 50 water molecules. For 50 water molecules, you need to have 50 oxygen atoms and 100 hydrogen atoms. If we simulate each atom using a thread, we require two hydrogen threads, and one oxygen thread to enter their critical sections, in order to generate one water molecule and have it counted.

In the following code, we firstly create 50 oxygen threads and 1 00 hydrogen threads. For protecting the oxygen thread’s critical section, we use a mutex, but for the hydrogen threads’ critical sections, we use a general semaphore that allows two threads to enter the critical section simultaneously.

For signaling purposes, we use POSIX barriers, but since barriers are not implemented in Apple systems, we need to implement them using mutexes and condition variables. The following is the code:

#include 
#include 
#include 
#include 
#include  // For errno and strerror function
// The POSIX standard header for using pthread library
#include 
// Semaphores are not exposed through pthread.h
#include 
#ifdef __APPLE__
// In Apple systems, we have to simulate the barrier
functionality.
pthread_mutex_t barrier_mutex;
pthread_cond_t barrier_cv;
unsigned int barrier_thread_count;
unsigned int barrier_round;
unsigned int barrier_thread_limit;

void barrier_wait(){
    pthread_mutex_lock(&barrier_mutex);
    barrier_thread_count++;
    if (barrier_thread_count >= barrier_thread_limit) {
       barrier_thread_count = 0;
       barrier_round++;
       pthread_cond_broadcast(&barrier_cv);
    } else {
       unsigned int my_round = barrier_round;
       do {
           pthread_cond_wait(&barrier_cv, &barrier_mutex);
       } while (my_round == barrier_round);
    }
    pthread_mutex_unlock(&barrier_mutex);
}
#else
// A barrier to make hydrogen and oxygen threads synchronized
pthread_barrier_t water_barrier;
#endif

// A mutex in order to synchronize oxygen threads
pthread_mutex_t    oxygen_mutex;
// A general semaphore to make hydrogen threads synchronized
sem_t*             hydrogen_sem;
// A shared integer counting the number of made water molecules
unsigned int       num_of_water_molecules;
void* hydrogen_thread_body(void* arg) {
    // Two hydrogen threads can enter this critical section
    sem_wait(hydrogen_sem);
    // Wait for the other hydrogen thread to join
#ifdef __APPLE__
    barrier_wait();
#else
    pthread_barrier_wait(&water_barrier);
#endif
    sem_post(hydrogen_sem);
    return NULL;
}

void* oxygen_thread_body(void* arg) {
    pthread_mutex_lock(&oxygen_mutex);  
    // Wait for the hydrogen threads to join
#ifdef __APPLE__
    barrier_wait();
#else
    pthread_barrier_wait(&water_barrier);
#endif
    num_of_water_molecules++;
    pthread_mutex_unlock(&oxygen_mutex);
    return NULL;
}
int main(int argc, char** argv) {

    num_of_water_molecules = 0;

    // Initialize oxygen mutex
    pthread_mutex_init(&oxygen_mutex, NULL);
    // Initialize hydrogen semaphore
#ifdef __APPLE__
    hydrogen_sem = sem_open(“hydrogen_sem”,
             O_CREAT | O_EXCL, 0644, 2);
#else
    sem_t local_sem;
    hydrogen_sem = &local_sem;
    sem_init(hydrogen_sem, 0, 2);
#endif

    // Initialize water barrier
#ifdef __APPLE__
    pthread_mutex_init(&barrier_mutex, NULL);
    pthread_cond_init(&barrier_cv, NULL);
    barrier_thread_count = 0;
    barrier_thread_limit = 0;
    barrier_round = 0;
#else
    pthread_barrier_init(&water_barrier, NULL, 3);
#endif

    // For creating 50 water molecules, we need 50 oxygen atoms and
    // 100 hydrogen atoms
    pthread_t thread[150];
    // Create oxygen threads
    for (int i = 0; i < 50; i++) {
        if (pthread_create(thread + i, NULL,
            oxygen_thread_body, NULL)) {
        printf(“Couldn’t create an oxygen thread.\n”);
        exit(1);
        }
    }
    // Create hydrogen threads
    for (int i = 50; i < 150; i++) {
        if (pthread_create(thread + i, NULL,
            hydrogen_thread_body, NULL)) {
        printf(“Couldn’t create an hydrogen thread.\n”);
        exit(2);
        }
    }
    printf(“Waiting for hydrogen and oxygen atoms to react …\n”);
    // Wait for all threads to finish
    for (int i = 0; i < 150; i++) {
        if (pthread_join(thread[i], NULL)) {
            printf(“The thread could not be joined.\n”);
            exit(3);
        }
    }
    printf(“Number of made water molecules: %d\n”,
        num_of_water_molecules);
#ifdef __APPLE__
    sem_close(hydrogen_sem);
#else
    sem_destroy(hydrogen_sem);
#endif

    return 0;
}

At the beginning of the code, there are several lines that are surrounded by #ifdef __APPLE__ and #endif. These lines are only compiled in Apple systems. These lines are mainly the implementation and variables required for simulating POSIX barrier behavior. In other POSIX-compliant systems other than Apple, we use an ordinary POSIX barrier.

As part of several global variables defined in the preceding code, we have declared the mutex oxygen_mutex which is supposed to protect the oxygen threads’ critical sections. At each time, only one oxygen thread (or oxygen atom) can enter the critical section.

Then in its critical section, an oxygen thread waits for two other hydrogen threads to join and then it continues to increment the water molecule counter. The increment happens within the oxygen’s critical section.

To elaborate more on the things that happen inside the critical sections, we need to explain the role of the general semaphore. In the preceding code, we have also declared the general semaphore, hydrogen_sem which is supposed to protect hydrogen threads’ critical sections. At each time, only a maximum of two hydrogen threads can enter their critical sections, and they wait on the barrier object shared between the oxygen and hydrogen threads.

When the number of waiting threads on the shared barrier object reaches two, it means that we have got one oxygen and two hydrogens, and then voilà: a water molecule is made, and all waiting threads can continue. Hydrogen threads exit immediately, but the oxygen thread exists only after incrementing the water molecules counter.

We close this section with this last note. In this example, we used the pthread_cond_broadcast function when implementing the barriers for Apple systems. It signals all threads, waiting on the barrier’s condition variable, that are supposed to continue after having other threads joining them.

Summary:

Concurrency is an important aspect of programming and maintaining data integrity is the key. In this article, we explored the general semaphore as a part of thread synchronization in multi-threaded programming. Further, we explored POSIX condition variables to wait for a specific condition, various types of locks together with mutexes and condition variables, the memory structure of a thread and how this structure can affect memory visibility in a multi-core system and much more…

About the author:

Kamran Amini is a senior professional specialized in embedded and kernel development. He has worked for numerous Iranian well-known companies. In 2017, he moved to Europe to work as a senior architect and engineer for highly reputable companies such as Jeppesen, Adecco, TomTom, and ActiveVideo Networks. While residing in Amsterdam, he worked on his first book, Extreme C, published by PacktPub. His main areas of interest are computation theory, distributed systems, machine learning, information theory, and quantum computation. Parallel to his professional career, he is studying Astronomy and Planetary sciences.

Role of Declaration Files in the Implementation of TypeScript

Packt — Thu, 20 May 2021 10:56:31 GMT

In this article, we will look at how we can enhance a JavaScript library and add syntactic sugar through declaration files.

One of the most appealing facets of JavaScript development is the wealth of external JavaScript libraries that have already been published, are tried and tested, and are available for re-use.

This article is an excerpt from the book, Mastering TypeScript, 4th Edition by Nathan Rozentals — A comprehensive guide to understand TypeScript language and its latest features.

Declaration files

A declaration file is a special type of file used by the TypeScript compiler. It is only used during the compilation step and is used as a sort of reference file to describe JavaScript. Declaration files are similar to the header files used in C or C++ or the interfaces used in Java. They simply describe the structure of available functions and properties but do not provide an implementation. In this section of the chapter, we will take a look at these declaration files, what they are, and how to write them.

Declaration file typing

As we know, declaration files use the declare and module keywords to define objects and namespaces. We have also seen that we can use interfaces in the same way that we do within TypeScript, in order to define custom types for variables. Declaration files allow us to use the same syntax that we would in TypeScript to describe types. These types can be used everywhere types are used in normal TypeScript, including function overloading, type unions, classes, and optional properties. Let’s take a quick look at these techniques, with a few simple code samples, to illustrate this feature further. This section will cover:

Function overloading
Nested namespaces
Classes
Static properties and functions
Abstract classes
Generics
Conditional types and inference

We’ll begin by taking a look at function overloading.

Function overloading

Declaration files allow for function overloads, where the same function can be declared with different arguments, as follows:

declare function trace(arg: string | number | boolean);
declare function trace(arg: { id: number; name: string });

Here, we have a function named trace that is declared twice: once with a single parameter named arg, of type string or number or boolean, and once with the same arg parameter, which is a custom type. This overloaded declaration allows for all of the following valid code:

trace(“trace with string”);
trace(true);
trace(1);
trace({ id: 1, name: “test” });

Here, we have exercised the various combinations of arguments that are allowed by our function overrides.

Nested namespaces

Declaration files allow for module names to be nested. This in turn translates to nested namespaces, as follows:

declare module FirstNamespace {
   module SecondNamespace {
      module ThirdNamespace {
          function log(msg: string);
      }
   }
}

Here, we have declared a module named FirstNamespace that exposes a module named SecondNamespace, which in turn exposes a third module named ThirdNamespace. The ThirdNamespace module defines a function named log. This declaration will result in all three namespaces needing to referenced in order to call the log function, as follows:

FirstNamespace.SecondNamespace.ThirdNamespace.log(“test”);

Here, we are explicitly referencing each named namespace in order to call the log function within the ThirdNamespace module.

Classes

Class definitions are specified in module definitions using the class keyword, as it would be in normal TypeScript files, as follows:

declare class MyModuleClass {
   public print(): void;
}

Here, we have declared a class named MyModuleClass that has a public print function that returns void. This class definition can then be used as follows:

let myClass = new MyModuleClass();
myClass.print();

Here, we are creating an instance of the MyModuleClass class that has been declared in our declaration file. We then are calling the print function of the class instance named myClass.

Declaring a class in a declaration file is very similar to defining an interface for it. We do not provide any implementations of the functions, we are only declaring to the TypeScript compiler that the class exists and what properties and functions are available to it.

Static properties and functions

In the same manner that we can mark a property or function as static in TypeScript, we can use the same syntax in a declaration file, as follows:

declare class MyModuleStatic {
    static print(): void;
    static id: number;
}

Here, we have declared a class named MyModuleStatic that has a static print function and a static id property. We can use these static properties and functions as follows:

MyModuleStatic.id = 10;
MyModuleStatic.print();

Here, we can see that the declaration of a static function or property follows the same usage rules as if we had defined it in TypeScript.

Abstract classes

Declaration files can define abstract classes and functions as follows:

declare abstract class MyModuleAbstract {
    abstract print(): void
}

Here, we have defined an abstract class named MyModuleAbstract that has a single function named print that has also been marked as abstract. We can use this abstract class declaration in a TypeScript file as follows:

class DerivedFromAbstract extends MyModuleAbstract {
    print() { }
}

Here, we have defined a class named DerivedFromAbstract that extends the MyModuleAbstract class from our declaration file.

Note that we will also need to provide an implementation of the print function within this class definition, as the print function has been marked as an abstract function in the class declaration.

Generics

Declaration files allow generic syntax to be used, as follows:

declare function sort
    (input: Array): Array { }

Here, we are declaring a function named sort that is using generic syntax to specify the type of T to be either a number or a string, or both. This sort function has a single parameter named input that is an array of type T and returns an array of type T. This declaration will allow the following usage:

let sortedStringArray = sort([“first”, “second”]);
let sortedNumericArray = sort([1, 2, 3]);

Here, we have defined a variable named sortedStringArray to hold the return value of a call to the sort function, where we pass in an array of strings as the only argument. The type of the sortedStringArray variable will be Array. Next, we define a variable named sortedNumericArray to hold the return value of another call to the sort function, where we pass in an array of numbers as the only argument. The sortedNumericArray variable will be of type Array.

Conditional types

Declaration files can also define conditional types and distributed conditional types, as follows:

declare type stringOrNumberOrBoolean =
    T extends string ? string :
    T extends number ? number :
    T extends boolean ? boolean : never;

Here, we have declared a type named stringOrNumberOrBoolean that is using generic syntax to define a type of T. If T extends string, then the type will be string. If T extends number, then the type will be number. If the type of T extends boolean, then the type will be boolean. If T does not extend any of these types, then the type will be never.

We can now use this distributed conditional type as follows:

type myNever = stringOrNumberOrBoolean<[string,number]>;

Here, we have defined a type named myNever that is the result of the distributed conditional type named stringOrNumberOrBoolean, and has defined a type of T as a tuple of string and number. As this tuple does not match any of the types we are checking for, the type of the myNever variable will be never.

Conditional type inference

Declaration files allow for conditional type inference, as can be seen from the following example:

declare type inferFromPropertyType =
    T extends { id: infer U } ? U : never;

Here, we have a type named inferFromPropertyType that will infer the type named U from the property named id of the type T. If the id property does not exist, then the type will be never. We can now use this type as follows:

type myString = inferFromPropertyType<{ id: string }>;
type myNumber = inferFromPropertyType<{ id: number }>;

Here, we have defined two types, named myString and myNumber, that are using the inferred conditional type named inferFromPropertyType. As the type of the id property is a string in the first line, then myString will be of type string. The type of the id property in the second line of code is number, and therefore myNumber will be of type number.

Declaration file summary

What we have seen in this article is that the type rules and typing techniques that TypeScript provides can all be used within declaration files. The purpose of a declaration file is to tell the TypeScript compiler ahead of time what the structure of a JavaScript library looks like. We have seen that we can use all of the TypeScript keywords and language features within a declaration file.

About the author

Nathan Rozentals has been writing commercial software for over 30 years, in C, C++, Java, and C#. He picked up TypeScript within a week after its initial release in October 2012 and realized how much TypeScript could help when writing JavaScript.

Nathan’s TypeScript solutions now control User Interfaces in IoT devices, run as stand-alone applications for Point-of-Sale solutions, provide complex application configuration websites, and are used for mission-critical server APIs.

This article is an excerpt from the book, Mastering TypeScript, 4th Edition by Nathan Rozentals — A comprehensive guide to understand TypeScript language and its latest features. If you like the article then you will love the book too.

Role of Declaration Files in the Implementation of TypeScript was originally published in Javarevisited on Medium, where people are continuing the conversation by highlighting and responding to this story.

Essential Statistics for Data Assessment

Packt — Tue, 11 May 2021 14:35:35 GMT

In this article we are going to learn about essential statistics for data assessment, also often referred to as descriptive statistics. Descriptive statistics provide simple, quantitative summaries of datasets, usually combined with graphics. As we shall shortly see, they can demonstrate the tendency to centralization, provide measures of the variability of features, and much more besides.

Note that there is another kind of statistics, known as inferential statistics, which tries to learn information from the distribution of the population that the dataset was generated or sampled from. Here, we assume the data covers a whole population rather than a subset sampled from a distribution.

The topics we shall cover include:

Classifying numerical and categorical variables
Understanding mean, median, and mode
Learning about variance, standard deviation, percentiles, and skewness
Knowing how to handle categorical variables and mixed data types
Using bivariate and multivariate descriptive statistics.

Practical examples are provided using Python, which is probably the most popular programming language for data science.

Classifying numerical and categorical variables

Descriptive statistics are all about variables. You must know what you are describing to define corresponding descriptive statistics.

A variable is sometimes referred to as a feature or attribute in other literature. They all mean the same thing: a single column in a tabulated dataset.

In this section, you will examine the two most important variable types, numerical and categorical, and learn to distinguish between them. Categorical variables are discrete and usually represent a classification property of an item. Numerical variables are continuous and are descriptive quantitatively. Descriptive statistics that can be applied to one kind of variable may not be applicable to the other kind, and hence distinguishing between them precedes analytics.

Distinguishing between numerical and categorical variables

To understand the differences between the two types of variables, I will use the example of the population estimates dataset released by the United States Department of Agriculture. It contains the estimated population data at county level for the United States from 2010 to 2018. You can obtain the data from the official website, https://www.ers.usda.gov/data-products/county-level-data-sets/download-data/.

The following Python code snippet loads the data and examines the first several rows:

import pandas as pd
df = pd.read_excel("PopulationEstimates.xls",skiprows=2)
df.head(8)

The output is a table with more than 140 columns. Here is a screenshot showing the beginning columns:

Figure 1 - First 6 columns of the dfad() output

In the dataset there is a variable called Rural-urban_Continuum Code_2013. It takes integer values. This leads to pandas auto-interpreting this variable; pandas auto-interprets it as numerical. Instead, however, the variable is actually categorical.

Should you always trust libraries?

Don’t always trust the functions that Python libraries give you. They may be wrong, and the developer, who is you, has to make the final decision.

After some research, we found the variable description on this page: https://www.ers.usda.gov/data-products/rural-urban-continuum-codes/.

According to the code standard published in 2013, the Rural-urban_Continuum_Code_2013 variable indicates how urbanized an area is.

The meaning of Rural-urban_Continuum Code_2013 is shown in Figure 2.

Figure 2 - Interpretation of Rural-urban_Continuum Code_2013

Note

Pandas makes intelligent auto-interpretations of variable types, but oftentimes it is wrong. It is up to the data scientist to investigate the exact meaning of the variable type and then change it.

Many datasets use integers to represent categorical variables. Treating them as numerical values may result in serious consequences in terms of downstream tasks such as machine learning, mainly because artificial distances between numerical values will be introduced.

On the other hand, numerical variables often have a direct quantitative meaning. For example, R_NET_MIG_2013 means the rate of net immigration in 2013 for a specific area. A histogram plot of this numerical variable gives a more descriptive summary of immigration trends in the States, but it makes little sense plotting the code beyond simple counting.

Let’s check the net immigration rate for the year 2013 with the following code snippet:

plt.figure(figsize=(8,6))
plt.rcParams.update({'font.size': 22})
plt.hist(df["R_NET_MIG_2013"],bins=np.linspace(np.nanmin(df["R_
NET_MIG_2013"]),np.nanmax(df["R_NET_MIG_2013"]),num=100))
plt.title("Rate of Net Immigration Distribution for All
Records, 2013");

The result appears as follows.

Figure 3 - Distribution of the immigration rate for all records in datasets

Here are the observations drawn from Figure 3:

With both categorical and numerical variables, structures can be introduced to construct special cases. A typical example is date or time. Depending on the scenario, date and time can be treated as either a categorical variable or as a numerical variable with a semi-continuous structure.
It is common to convert numerical variables to categorical variables on the basis of a number of rules. The rural-urban code is a typical example. Such a conversion is easy for conveying a first impression.

Now that we have learned how to distinguish between numerical and categorical variables, let’s move on to understanding a few essential concepts of statistics, namely mean, median, and mode.

Understanding mean, median, and mode

Mean, median, and mode describe aspects of the central tendency. Mean and median are only applicable to numerical variables, whereas mode is applicable to both categorical and numerical variables. In this section, we will focus on mean, median, and mode for numerical variables, as their numerical interactions usually convey interesting information.

Mean

Mean, or arithmetical mean measures the weighted center of a variable. Let’s use n to denote the total number of entries and as the index. The mean is given by the following expression:

Mean is influenced by the value of every entry in the population.

Let me give an example. The following code generates 1,000 random numbers from 0 to 1 uniformly, plots them, and calculates their mean:

import random
random.seed(2019)
plt.figure(figsize=(8,6))
rvs = [random.random() for _ in range(1000)]
plt.hist(rvs, bins=50)
plt.title("Histogram of Uniformly Distributed RV");

The resulting histogram plot appears as follows:

Figure 4 - Histogram distribution of uniformly distributed variables between 0 and 1

The mean is around 0.505477, pretty close to what we’d expect.

Median

Median measures the unweighted center of a variable. If there is an odd number of entries, the median takes the value of the central one. If there is an even number of entries, the median takes the value of the mean of the central two entries. The median may not be influenced by every entry’s value. On account of this property, the median is more robust or representative than the mean value. I will use the same set of entries as in previous sections as an example.

The following code calculates the median:

np.median(rvs)

The result is 0.5136755026003803. Now, I will be changing one entry to 1,000, which is 1,000 times larger than the maximal possible value in the dataset, and repeat the calculation:

rvs[-1]=1000
print(np.mean(rvs))
print(np.median(rvs))

The results are 1.5054701085937803 and 0.5150437661964872. The mean increased by roughly 1, but the median is robust.

The relationship between mean and median is usually interesting and worth investigating. Usually, the combination of a larger median and smaller mean indicates that there are more points on the bigger value side, but that an extremely small value also exists. The reverse is true when the median is smaller than the mean. We will demonstrate this with some examples later.

Mode

The mode of a set of values is the most frequent element in a set. It is evident in a histogram plot such that it represents the peak(s). If the distribution has only one mode, we call it unimodal. Distributions with two peaks that don’t have to have equal heights are referred to as bimodal.

Bimodals and bimodal distribution

Sometimes, the definition of bimodal is corrupted. The property of being bimodal usually refers to the property of having two modes, which, according to the definition of mode, requires the same height of peaks. However, the term bimodal distribution often refers to a distribution with two local maxima. Double-check your distribution and state the modes clearly.

The following code snippet demonstrates two distributions with unimodal and bimodal shapes respectively:

r1 = [random.normalvariate(0.5,0.2) for _ in range(10000)]
r2 = [random.normalvariate(0.2,0.1) for _ in range(5000)]
r3 = [random.normalvariate(0.8,0.2) for _ in range(5000)]

fig, axes = plt.subplots(1,2,figsize=(12,5))
axes[0].hist(r1,bins=100)
axes[0].set_title("Unimodal")
axes[1].hist(r2+r3,bins=100)
axes[1].set_title("Bimodal");

The resulting two subplots look as follows:

Figure 5 - Histogram of unimodal and bimodal datasets with one mode and two modes

So far, we have talked about mean, median, and mode, which are the first three statistics of a dataset. They are the start of almost all exploratory data analysis.

Learning about variance, standard deviation, quartiles, percentiles, and skewness

In the previous section, we studied the mean, median, and mode. They all describe, to a certain degree, the properties of the central part of the dataset. In this section, we will learn how to describe the spreading behavior of data.

Variance

Using the same notation, variance for the population is defined as follows:

Intuitively, the further away the elements are from the mean, the larger the variance. Here, I’ve plotted the histogram of two datasets with different variances. The one in the left sub-plot has a variance of 0.09 and the one in the right sub-plot has a variance of 0.009, i.e. 10 times smaller.

The following code snippet generates samples from the two distributions and plots them:

r1 = [random.normalvariate(0.5,0.3) for _ in range(10000)]
r2 = [random.normalvariate(0.5,0.1) for _ in range(10000)]

fig, axes = plt.subplots(1,2,figsize=(12,5))
axes[0].hist(r1,bins=100)
axes[0].set_xlim([-1,2])
axes[0].set_title("Big Variance")
axes[1].hist(r2,bins=100)
axes[1].set_title("Small Variance")
axes[1].set_xlim([-1,2]);

The results are as follows:

Figure 6 - Big and small variances with the same mean at 0.5

The following code snippet generates a scatter plot that will demonstrate the difference more clearly. The variable on the x axis spreads more widely:

plt.figure(figsize=(8,8))
plt.scatter(r1,r2,alpha=0.2)
plt.xlim(-1,2)
plt.ylim(-1,2)
plt.xlabel("Big Variance Variable")
plt.ylabel("Small Variance Variable")
plt.title("Variables With Different Variances");

The result looks as follows:

Figure 7 - Scatter plot of large-variance and small-variance variables

The spread in the x axis is significantly larger than the spread in the y axis, which indicates the differences in variance magnitude. A common mistake is not getting the range correct. Matplotlib will, by default, try to determine the ranges. You need to use a code such as plt.xlim() to force it, otherwise the result can be misleading.

Standard deviation

Standard deviation is the square root of the variance. It is used more commonly to measure the level of dispersion since it has the same unit as the original data. The formula for the standard deviation of a population is:

Standard deviation is extremely important in scientific graphing. A standard deviation is often plotted together with the data, and represents an estimate of variability.

For this article, I will be using the net immigration rate for Texas from 2011 to 2018 as an example. In the following code snippet, I will first extract the county-level data, append the means and standard deviations to a list, and then plot them at the end. The standard deviation is obtained using numpy.std() and the error bar is plotted using matplotlib.pyplot.errorbar():

dfTX = df[df["State"]=="TX"].tail(-1)

YEARS = [year for year in range(2011,2019)]
MEANS = []
STDS = []
for i in range(2011,2019):
    year = "R_NET_MIG_"+str(i)
    MEANS.append(np.mean(dfTX[year]))
    STDS.append(np.std(dfTX[year]))

plt.figure(figsize=(10,8))
plt.errorbar(YEARS,MEANS,yerr=STDS)
plt.xlabel("Year")
plt.ylabel("Net Immigration Rate");

The output appears as shown in the following figure:

Figure 8 - Net immigration rate across counties in Texas from 2011 to 2018

We can see in Figure 8 that although the net immigration in Texas is only slightly positive, the standard deviation is huge. Some counties may have a big positive net rate, while others may potentially suffer from the loss of human resources.

Quartiles

Quartiles are a special kind of quantile that divides data into a number of equal portions. For example, quartiles divide data into four equal parts with the 1/2 quartile as the median. Deciles and percentiles divide data into 10 and 100 equal parts, respectively.

The first quartile, also known as the lower quartile Q1 takes the value such that 25% of all the data lies below it. The second quartile is the median. The third quartile, Q3, is also known as the upper quartile and 25% of all values lie above it.

Quartiles are probably the most commonly used quantiles because they are associated with a statistical graph called a boxplot. Let’s use the same set of Texas net immigration data to study it.

The function in NumPy is quantile() and we specify a list of quantiles as an argument for the quantiles we want to calculate, as in the following single-line code snippet:

np.quantile(dfTX["R_NET_MIG_2013"],[0.25,0.5,0.75])

The output reads as follows:

plt.figure(figsize=(12,5))
plt.hist(dfTX["R_NET_MIG_2013"],bins=50,alpha=0.6)
for quartile in np.quantile(dfTX["R_NET_
MIG_2013"],[0.25,0.5,0.75]):
    plt.axvline(quartile,linestyle=':',linewidth=4)

As you can see from the following output, the vertical dotted lines indicate the three quartiles:

Figure 9 - Quartiles of the net immigration data in 2013

The lower and upper quartiles keep exactly 50% of the data values in between. Q3 — Q1 is referred to as the interquartile range called Interquartile Range (IQR) and it plays an important role in outlier detection. We will see more about this shortly.

Skewness

Skewness differs from the three measures of variability discussed in the previous subsections. It measures the direction the data takes and the extent to which the data distribution tilts. Skewness is given by the following equation:

Various definitions of skewness

The skewness we defined earlier is precisely referred to as Pearson’s first skewness coefficient. It is defined through the mode, but there are other definitions of skewness. For example, skewness can be defined through the median.

Skewness is unit-less. If the mean is larger than the mode, skewness is positive, and we say the data is skewed to the right. Otherwise, the data is skewed to the left.

Here is a code snippet that generates two sets of skewed data and plots them:

r1 = [random.normalvariate(0.5,0.4) for _ in range(10000)]
r2 = [random.normalvariate(0.1,0.2) for _ in range(10000)]
r3 = [random.normalvariate(1.1,0.2) for _ in range(10000)]

fig, axes = plt.subplots(1,2,figsize=(12,5))
axes[0].hist(r1+r2,bins=100,alpha=0.5)
axes[0].axvline(np.mean(r1+r2), linestyle=':',linewidth=4)
axes[0].set_title("Skewed To Right")
axes[1].hist(r1+r3,bins=100,alpha=0.5)
axes[1].axvline(np.mean(r1+r3),linestyle=':',linewidth=4)
axes[1].set_title("Skewed to Left");

The vertical dotted line indicates the position of the mean as follows:

Figure 10 - Skewness demonstration

Think about the problem of income inequality. Let’s say you have a plot of the histogram of the population with different amounts of wealth. A larger value just like where the x axis value indicates the amount of wealth and the y axis value indicates the portion of the population that falls into a certain wealth amount range. A larger x value means more wealth. A larger y value means a greater percentage of the population falls into that range of wealth possession. Positive skewness (the left subplot in Figure 10) means that even though the average income looks good, this may be driven up by a very small number of super-rich individuals when the majority of people earn a relatively small income. Negative skewness (the right subplot in Figure 10) indicates that the majority may have an income above the mean value, so there might be some very poor people who may need help.

Revisiting outlier detection

Now let’s use what we have learned to revisit the outlier detection problem.

The z-score, also known as the standard score, is a good criterion for detecting outliers. It measures the distance between an entry and the population mean, taking the population variance into consideration:

If the underlying distribution is normal, a situation where a z-score is greater than 3 or less than 0 only has a probability of roughly 0.27%. Even if the underlying distribution is not normal, Chebyshev’s theorem guarantees a strong claim such that at most 1/(k^2), where k is an integer, of the total population can fall outside k standard deviations.

As an example, the following code snippet generates 10,000 data points that follow a normal distribution:

random.seed(2020)
x = [random.normalvariate(1, 0.5) for _ in range(10000)]
plt.figure(figsize=(10,8))
plt.hist(x,bins=100,alpha=0.5);
styles = [":","--","-."]
for i in range(3):
    plt.axvline(np.mean(x) + (i+1)*np.std(x),
        linestyle=styles[i],
        linewidth=4)
    plt.axvline(np.mean(x) - (i+1)*np.std(x),
        linestyle=styles[i],
        linewidth=4)
plt.title("Integer Z values for symmetric distributions");

In the generated histogram plot, the dotted line indicates the location where z = ±1 . The dashed line indicates the location of z = ±2 . The dashed-dotted line indicates the location of z = ±3:

Figure 11 - Integer z value boundaries for normally distributed symmetric data

If we change the data points, the distribution will change, but the z-score criteria will remain valid. As you can see in the following code snippet, an asymmetric distribution is generated rather than a normal distribution:

x = [random.normalvariate(1, 0.5) + random.expovariate(2) for _
in range(10000)]

This produces the following output:

Figure 12 - Integer z value boundaries for asymmetric data

Note on the influence of extreme outliers

A drawback of the z-score is that the mean itself is also influenced by extreme outliers. The median can replace a mean to remove this effect.

We have covered several of the most important statistics to model variances in a dataset. In the next section, let’s work on the data types of features.

Knowing how to handle categorical variables and mixed data types

Categorical variables usually have simpler structures or descriptive statistics than continuous variables. Here, we introduce frequencies and proportions and talk about some interesting descriptive statistics examples when converting continuous variables to categorical ones.

We have covered several of the most important statistics to model variances in a dataset. In the next section, let’s work on the data types of features.

Knowing how to handle categorical variables and mixed data types

Frequencies and proportions

When we discussed the mode for categorical variables, we introduced Counter, which outputs a dictionary structure whose key-value pair is the element-counting pair. The following is an example of a counter:

Counter({2.0: 394, 3.0: 369, 6.0: 597, 1.0: 472, 9.0: 425, 7.0:
434, 8.0: 220, 4.0: 217, 5.0: 92})

The following code snippet illustrates frequency as a bar plot where the absolute values of counting become intuitive:

counter = Counter(df["Rural-urban_Continuum Code_2013"].
dropna())
labels = []
x = []
for key, val in counter.items():
    labels.append(str(key))
    x.append(val)
plt.figure(figsize=(10,8))
plt.bar(labels,x)
plt.title("Bar plot of frequency");

This produces a bar plot like the following:

Figure 13 - Bar plot of rural-urban continuum code

For proportionality, simply divide each count by the summation of counting, as shown in the following code snippet:

x = np.array(x)/sum(x)

The shape of the bar plot remains the same, but the y axis ticks change. To better check the relative size of components, I have plotted a pie plot with the help of the following code snippet:

plt.figure(figsize=(10,10))
plt.pie(x=x,labels=labels,)
plt.title("Pie plot for rural-urban continuum code");

This creates a nice pie chart as follows:

Figure 14 - Pie plot of rural-urban continuum code

It becomes evident that code 2.0 contains about twice as many samples as code 8.0 does.

Unlike the mean and median, categorical data does have a mode. We are going to reuse the same data:

Counter(df["Rural-urban_Continuum Code_2013"].dropna())

The output reads as follows:

Counter({2.0: 394, 3.0: 369, 6.0: 597, 1.0: 472, 9.0: 425, 7.0:
434, 8.0: 220, 4.0: 217, 5.0: 92})

The mode is 6.0.

Note

The mode means that the counties with urban populations of 2,500 to 19,999 adjacent to a metro area are most prevalent in the United States, and not the number 6.0.

Transforming a continuous variable to a categorical one

Occasionally, we may need to convert a continuous variable to a categorical one. Let’s take lifespan as an example. The 80+ age group is supposed to be very small. Each of them will represent a negligible data point in classification tasks. If they can be grouped together, the noise introduced by the sparsity of this age group’s individual points will be reduced.

A common way to perform categorization is to use quantiles. For example, quartiles will divide the datasets into four parts with an equal number of entries. This avoids issues such as data imbalance.

For example, the following code indicates the cut-offs for the categorization of net immigration rate, a continuous variable:

series = df["R_NET_MIG_2013"].dropna()
quantiles = np.quantile(series,[0.2*i for i in range(1,5)])
plt.figure(figsize=(10,8))
plt.hist(series,bins=100,alpha=0.5)
plt.xlim(-50,50)
for i in range(len(quantiles)):
    plt.axvline(quantiles[i],linestyle=":", linewidth=4)
plt.title("Quantiles for net immigration data");

As you can see in the following output, the dotted vertical lines split the data into 5 equal sets, which are hard to spot with the naked eye. I truncated the x axis to select the part between -50 and 50. The result looks as follows:

Figure 15 - Quantiles for the net immigration rate

Note on the loss of information

Categorization destroys the rich structure in continuous variables. Only use it when you absolutely need to.

Using bivariate and multivariate descriptive statistics

In this section, we briefly talk about bivariate descriptive statistics. Bivariate descriptive statistics apply two variables rather than one. We are going to focus on correlation for continuous variables and cross-tabulation for categorical variables.

Covariance

The word covariance is often incorrectly used as correlation. However, there are a number of fundamental differences. Covariance usually measures the joint variability of two variables, while correlation focuses more on the strength of variability. Correlation coefficients have several definitions in different use cases. The most common descriptive statistic is the Pearson correlation coefficient. We will also be using it to describe the covariance of two variables. The correlation coefficient for variables x and y from a population is defined as follows:

Let’s first examine the expression’s sign. The coefficient becomes positive when x is greater than its mean and y is also greater than its own mean.

Another case is when x and y are both smaller than their means, respectively. The products sum together and then get normalized by the standard deviation of each variable. So, a positive coefficient indicates that x and y vary jointly in the same direction. You can make a similar argument about negative coefficients.

In the following code snippet, we select the net immigration rates for counties in Texas as our datasets and use the corr() function to inspect the correlation coefficient across years:

corrs = dfTX[['R_NET_MIG_2011','R_NET_MIG_2012', 'R_NET_
MIG_2013', 'R_NET_MIG_2014', 'R_NET_MIG_2015','R_NET_MIG_2016',
'R_NET_MIG_2017', 'R_NET_MIG_2018']].corr()

The output is a so-called correlation matrix whose diagonal elements are the self- correlation coefficients, which are just 1:

Figure 16 - Correlation matrix for the net immigration rate

A good way to visualize this matrix is to use the heatmap() function from the Seaborn library. The following code snippet generates a nice heatmap:

import seaborn as sns
plt.figure(figsize=(10,8))
plt.rcParams.update({'font.size': 12})
sns.heatmap(corrs,cmap="YlGnBu");

The result is as follows:

Figure 17 - Heatmap of a correlation matrix for net immigration rates in Texas

We do see an interesting pattern that odd years correlate with one another more strongly, and even years correlate with each other more strongly, too. However, that is not the case between even- and odd-numbered years. Perhaps there is a 2-year cyclic pattern and the heatmap of the correlation matrix just helped us discover it.

Cross-tabulation

Cross-tabulation can be treated as a discrete version of correlation detection for categorical variables. It helps derive innumerable insights and sheds light on downstream task designs.

Here is an example. I am creating a list of weather information and another list of a golfer’s decisions on whether to go golfing. The crosstab() function generates the following table:

weather = ["rainy","sunny","rainy","windy","windy",
           "sunny","rainy","windy","sunny","rainy",
           "sunny","windy","windy"]
golfing = ["Yes","Yes","No","No","Yes","Yes","No","No",
           "Yes","No","Yes","No","No"]
dfGolf = pd.DataFrame({"weather":weather,"golfing":golfing})
pd.crosstab(dfGolf.weather, dfGolf.golfing, margins=True)

Figure 18 - Cross-tabulation for golfing decisions

As you can see, the columns and rows give the exact counts, which are identified by the column name and row name. For a dataset with a limited number of features, this is a handy way to inspect imbalance or bias.

We can tell that the golfer goes golfing if the weather is sunny, and that they seldom go golfing on rainy or windy days.

In conclusion

In this article, we’ve explained what descriptive statistics are and what they can tell you. You have seen how to extract information from your datasets and use descriptive statistics to make quantitative judgments. Hopefully, it is now clear that an understanding of the different descriptive statistics, knowledge of when to use them, and the practical ability to derive them from your data, are indispensable items in the data scientist’s toolkit.

Equipped with this knowledge, you can reinforce your understanding of data science and data analysis from a statistical perspective and extract meaningful insights from your data using Python programming. Find out how with Rongpeng Li’s book Essential Statistics for Non-STEM Data Analysts.

Implementing Terminal I/O in Rust

Packt — Wed, 05 May 2021 16:23:21 GMT

Rust is a modern, open source system programming language that promises the best of three worlds: the type safety of Java; the speed, expressiveness, and efficiency of C++; and memory safety without a garbage collector. In this article, we look at building terminal-based applications in Rust.

Terminal applications are an integral part of many software programs, including games, text editors, and terminal emulators. For developing these types of programs, it helps to understand how to build customized terminal interface-based applications. We will review the basics of how terminals work, and then look at how to perform various types of actions on a terminal, such as setting colors and styles, performing cursor operations (such as clearing and positioning), and working with keyboard and mouse inputs.

We will cover the following topics:

Introducing terminal I/O fundamentals
Working with the terminal UI (size, color, styles) and cursors
Processing keyboard inputs and scrolling
Processing mouse inputs

The bulk of the article will be dedicated to explaining these concepts through a practical example. We will build a mini text viewer that will demonstrate key concepts of working with terminals. The text viewer will be able to load a file from disk and display its contents on the terminal interface. It will also allow a user to scroll through the contents using the various arrow keys on the keyboard, and display information on the header and footer bar.

Technical requirements

The Git repo for the code in this article can be found at https://github.com/PacktPublishing/Practical-System-Programming-for-Rust-Developers/tree/master/Chapter07/tui.

For those working on the Windows platform, a virtual machine needs to be installed, as the third-party crate used for terminal management does not support the Windows platform (at the time of writing). It is recommended to install a virtual machine such as VirtualBox or equivalent running Linux for working with the code. Instructions to install VirtualBox can be found at https://www.virtualbox.org.

For working with terminals, Rust provides several features to read keypresses and to control standard input and output for a process. When a user types characters in the command line, the bytes generated are available to the program when the user presses the Enter key. This is useful for several types of programs. But for some types of programs, such as games or text editors, which require more fine-grained control, the program must process each character as it is typed by the user, which is also known as raw mode. There are several third-party crates available that make raw mode processing easy. We will be using one such crate, Termion.

Introducing terminal I/O fundamentals

In this section, we’ll cover the key characteristics of terminals, see an overview of the Termion crate, and define the scope of what we will be building in this project.

Characteristics of terminals

Terminals are devices with which users can interact with a computer. Using a terminal, a user can get command-line access to interact with the computer’s operating system. A shell typically acts as the controlling program to drive the terminal on one hand and the interface with the operating system on the other hand.

Originally, UNIX systems were accessed using a terminal (also called a console) connected to a serial line. These terminals typically had a 24 x 80 row x column character-based interface, or, in some cases, had rudimentary graphics capabilities. In order to perform operations on the terminal, such as clearing the screen or moving the cursor, specific escape sequences were used.

There are two modes in which terminals can operate:

Canonical mode: In canonical mode, the inputs from the user are processed line by line, and the user has to press the Enter key for the characters to be sent to the program for processing.
Non-canonical or raw mode: In raw mode, terminal input is not collected into lines, but the program can read each character as it is typed by the user.

Terminals can be either physical devices or virtual devices. Most terminals today are pseudo-terminals, which are virtual devices that are connected to a terminal device on one side, and to a program that drives the terminal device on the other end. Pseudo-terminals help us write programs where a user on one host machine can execute a terminal-oriented program on another host machine using network communications. An example of a pseudo-terminal application is SSH (Secure Shell Protocol), which allows a user to log in to a remote host over a network.

Terminal management includes the ability to perform the following things on a terminal screen:

Color management: Setting various foreground and background colors on the terminal and resetting the colors to default values.
Style management: Setting the style of text to bold, italics, underline, and so on.
Cursor management: Setting the cursor at a particular position, saving the current cursor position, showing and hiding a cursor, and other special features, such as blinking cursors.
Event handling: Listening and responding to keyboard and mouse events.
Screen handling: Switching from main to alternate screens and clearing the screen.
Raw mode: Switching a terminal to raw mode.

In this article, we will use a combination of the Rust standard library and the Termion crate to develop a terminal-oriented application. Let’s begin by looking at the Termion crate.

The Termion crate

The Termion crate provides the features listed in the previous section, while also providing the user with easy-to-use command-line interfaces (CLIs).

Why use an external crate for terminal management?

While it is technically possible to work at the byte level using the Rust standard library, it is cumbersome. External crates such as Termion help us group individual bytes to keypresses, and also implement many of the commonly used terminal management functions, which allows us to focus on the higher level, user-directed functionality.

Let’s discuss a few terminal management features of the Termion crate. The official documentation of the crate can be found at https://docs.rs/termion/.

The Termion crate has the following key modules:

cursor: For moving cursors
event: For handling key and mouse events
raw: To switch the terminal to raw mode
style: To set various styles on text
clear: To clear the entire screen or individual lines
color: To set various colors to text
input: To handle advanced user input
scroll: To scroll across the screen

To include the Termion crate, start a new project and add the following entry to cargo.toml:

[dependencies]
termion = "1.5.5"

A few examples of Termion usage are shown through code snippets here:

To get the terminal size, use the following:

termion::terminal_size()

To set the foreground color, use the following:

println!(“{}”, color::Fg(color::Blue));

To set the background color and then reset the background color to the original state, use the following:

println!(
    “{}Background{} “,
    color::Bg(color::Cyan),
    color::Bg(color::Reset)
);

To set bold style, use the following:

println!(
    “{}You can see me in bold?”,
    style::Bold
);

To set the cursor to a particular position, use the following:

termion::cursor::Goto(5, 10)

To clear the screen, use the following:

print!(“{}”, termion::clear::All);

We will use these terminal management features in a practical example in the upcoming sections. Let’s now define what we are going to build.

What will we build?

We will develop a mini text viewer application. This application provides a terminal text interface to load a document from a directory location and view the document. The user can scroll through the document using keyboard keys. We’ll build this project progressively over multiple iterations of code.

Figure 1 shows the screen layout of what we will build:

Figure 1 - Text viewer screen layout

There are three components in the terminal interface of the text viewer:

Header bar: This contains the title of the text editor.
Text area: This contains the lines of text to be displayed.
Footer bar: This displays the position of the cursor, the number of lines of text in the file, and the name of the file being displayed.

The text viewer will allow the user to perform the following actions:

Users can provide a filename as a command-line argument to display. This should be a valid filename that already exists. If the file does not exist, the program will display an error message and exit.
The text viewer will load the file contents and display them on the terminal. If the number of lines in a file is more than the terminal height, the program will allow the user to scroll through the document, and repaint the next set of lines.
Users can use the up, down, left, and right keys to scroll through the terminal.
Users can press Ctrl + Q to exit the text viewer.

A popular text viewer would have a lot more features, but this core scope provides an adequate opportunity for us to learn about developing a terminal-oriented application in Rust.

In this section, we’ve learned what terminals are and what kinds of features they support. We also saw an overview of how to work with the Termion crate and defined what we will be building as part of the project. In the next section, we’ll develop the first iteration of the text viewer.

Working with the terminal UI (size, color, styles) and cursors

In this section, we will build the first iteration of the text viewer. At the end of this section, we will have a program that will accept a filename from the command line, display its contents, and display a header and footer bar. We will use a Termion crate to set the color and style, get the terminal size, position the cursor at specific coordinates, and clear the screen.

The code in this section is organized as follows:

Writing data structures and the main() function
Initializing the text viewer and getting the terminal size
Displaying a document and styling the terminal color, styles, and cursor position
Exiting the text viewer

Let's start with data structures and the main() function of the text viewer.

Writing data structures and the main() function

In this section, we’ll define the data structures needed to represent the text viewer in memory. We’ll also write the main() function, which coordinates and invokes various other functions:

1. Create a new project and switch to the directory with the following command:

cargo new tui && cd tui

Here, tui stands for terminal user interface. Create a new file called textviewer1.rs under src/bin.

2. Add the following to cargo.toml:

[dependencies]
termion = “1.5.5”

3. Let’s first import the required modules from the standard library and the Termion crate:

use std::env::args;
use std::fs;
use std::io::{stdin, stdout, Write};
use termion::event::Key;
use termion::input::TermRead;
use termion::raw::IntoRawMode;
use termion::{color, style};

4. Let’s next define the data structures to represent a text viewer:

struct Doc {
    lines: Vec,
}
#[derive(Debug)]
struct Coordinates {
    pub x: usize,
    pub y: usize,
}
struct TextViewer {
    doc: Doc,
    doc_length: usize,
    cur_pos: Coordinates,
    terminal_size: Coordinates,
    file_name: String,
}

This code shows three data structures defined for the text viewer:

The document that will be displayed in the viewer is defined as a Doc struct, which is a vector of strings.
To store cursor position x and y coordinates and to record the current size of the terminal (the total number of rows and columns of characters), we have defined a Coordinates struct.
The TextViewer struct is the main data structure representing the text viewer. The number of lines contained in the file being viewed is captured in the doc_length field. The name of the file to be shown in the viewer is recorded in the file_name field.

5. Let’s now define the main() function, which is the entry point for the text viewer application:

fn main() {
    //Get arguments from command line
    let args: Vec = args().collect();
    if args.len() < 2 {
        println!(“Please provide file name
            as argument”);
        std::process::exit(0);
    }
    //Check if file exists. If not, print error
    // message and exit process
    if !std::path::Path::new(&args[1]).exists() {
        println!(“File does not exist”);
        std::process::exit(0);
    }
    // Open file & load into struct
    println!(“{}”, termion::cursor::Show);
    // Initialize viewer
    let mut viewer = TextViewer::init(&args[1]);
    viewer.show_document();
    viewer.run();
}

The main() function accepts a filename as a command-line parameter and exits the program if the file does not exist. Furthermore, if a filename is not provided as a command-line parameter, it displays an error message and exits the program.

6. If the file is found, the main() function does the following:

It first calls the init() method on the TextViewer struct to initialize the variables.

Then, it invokes the show_document() method to display the contents of the file on the terminal screen.

Lastly, the run() method is called, which waits for user inputs to the process. If the user presses Ctrl + Q, the program exits.

7. We will now write three method signatures — init(), show_document(), and run(). These three methods should be added to the impl block of the TextViewer struct, as shown:

impl TextViewer {
    fn init(file_name: &str) -> Self {
        //...
    }
    fn show_document(&mut self) {
        // ...
    }
    fn run(&mut self) {
        // ...
    }
}

So far, we’ve defined the data structures and written the main() function with placeholders for the other functions. In the next section, let’s write the function to initialize the text viewer.

Initializing the text viewer and getting the terminal size

When a user starts the text viewer with a document name, we have to initialize the text viewer with some information and perform startup tasks. This is the purpose of the init() method.

Here is the complete code for the init() method:

fn init(file_name: &str) -> Self {
    let mut doc_file = Doc { lines: vec![] };                   <1>    
    let file_handle = fs::read_to_string(file_name)
        .unwrap();                                              <2>
    for doc_line in file_handle.lines() {                       <3>
        doc_file.lines.push(doc_line.to_string());
    }
    let mut doc_length = file_handle.lines().count();           <4>

    let size = termion::terminal_size().unwrap();               <5>
    Self {                                                      <6>
        doc: doc_file,
        cur_pos: Coordinates {
            x: 1,
            y: doc_length,
        },
        doc_length: doc_length,
        terminal_size: Coordinates {
            x: size.0 as usize,
            y: size.1 as usize,
        },
        file_name: file_name.into(),
    }
}

The code annotations in the init() method are described here:

Initialize the buffer that is used to store the file contents.
Read the file contents as a string.
Read each line from the file and store it in the Doc buffer.
Initialize the doc_length variable with the number of lines in the file.
Use the termion crate to get the terminal size.
Create a new struct of the TextViewer type and return it from the init() method.

We’ve written the initialization code for the text viewer. Next, we’ll write the code to display the document contents on the terminal screen, and also display the header and footer.

Displaying a document and styling the terminal color, styles, and cursor position

We saw earlier the layout of the text viewer that we would like to build. There are three main parts of the text viewer screen layout — the header, the document area, and the footer. In this section, we’ll write the primary function and supporting function to display the contents as per the defined screen layout.

Let’s look at the show_document() method:

src/bin/text-viewer1.rs

fn show_document(&mut self) {
    let pos = &self.cur_pos;
    let (old_x, old_y) = (pos.x, pos.y);
    print!("{}{}", termion::clear::All,
        termion::cursor::Goto(1, 1));
    println!(
        "{}{}Welcome to Super text viewer\r{}",
        color::Bg(color::Black),
        color::Fg(color::White),
        style::Reset
    );
    for line in 0..self.doc_length {
        println!("{}\r", self.doc.lines[line as usize]);
    }

    println!(
        "{}",
        termion::cursor::Goto(0, (self.terminal_size.y - 2) as
            u16),
    );
    println!(
        "{}{} line-count={} Filename: {}{}",
        color::Fg(color::Red),
        style::Bold,
        self.doc_length,
        self.file_name,
        style::Reset
    );
self.set_pos(old_x, old_y);
}

The code annotations for the show_document() method are described here:

Store the current positions of the cursor x and y coordinates in temp variables. This will be used to restore the cursor position in a later step.
Using the Termion crate, clear the entire screen and move the cursor to row 1 and column 1 on the screen.
Print the header bar of the text viewer. A background color of black and a foreground color of white is used to print text.
Display each line from the internal document buffer to the terminal screen.
Move the cursor to the bottom of the screen (using the terminal size y coordinate) to print the footer.
Print the footer text in red and with bold style. Print the number of lines in the document and filename to the footer.
Reset the cursor to the original position (which was saved to temporary variable in step 1).

Let’s look at the set_pos() helper method used by the show_document() method:

src/bin/text-viewer1.rs

fn set_pos(&mut self, x: usize, y: usize) {
    self.cur_pos.x = x;
    self.cur_pos.y = y;
    println!(
        "{}",
        termion::cursor::Goto(self.cur_pos.x as u16,
            (self.cur_pos.y) as u16)
    );
}

This helper method synchronizes the internal cursor tracking field (the cur_pos field of the TextViewer struct) and the on-screen cursor position.

We now have the code to initialize the text viewer and to display the document on the screen. With this, a user can open a document in the text viewer and view its contents. But how does the user exit the text viewer? We’ll find out in the next section.

Exiting the text viewer

Let’s say that a key combination of Ctrl + Q will let the user exit the text viewer program. How can we implement this code?

To achieve this, we need a way to listen for user key strokes, and when a particular key combination is pressed, we should exit the program. As discussed earlier, we need to get the terminal into raw mode of operation, where each character is available for the program to evaluate, rather than wait for the user to press the Enter key. Once we get the raw characters, the rest of it becomes fairly straightforward. Let’s write the code to do this in the run() method, within the impl TextViewer block, as shown:

src/bin/text-viewer1.rs

fn run(&mut self) {
    let mut stdout = stdout().into_raw_mode().unwrap();
    let stdin = stdin();
    for c in stdin.keys() {
        match c.unwrap() {
            Key::Ctrl(‘q’) => {
                break;
          }
          _=> {}
        }
        stdout.flush().unwrap();
    }
}

In the code shown, we use the stdin.keys() method to listen for user inputs in a loop. stdout() is used to display text to the terminal. When Ctrl + Q is pressed, the program exits.

We can now run the program with the following:

cargo run --bin text-viewer1

Since we have not implemented scrolling yet, pass a filename to the program that has 24 lines or less of content (this is typically the default height of a standard terminal in terms of the number of rows). You will see the text viewer open up and the header bar, footer bar, and file contents printed to the terminal. Type Ctrl + Q to exit. Note that you have to specify the filename with the full file path as a command-line argument.

In this section, we learned how to get the terminal size, set the foreground and background colors, and apply bold style using the Termion crate. We also learned how to position the cursor onscreen at specified coordinates, and how to clear the screen.

In the next section, we will look at processing keystrokes for user navigation within the document displayed in the text editor and how to implement scrolling.

Processing keyboard inputs and scrolling

In the previous section, we built the first iteration of our text viewer terminal-oriented application. We were able to display a file with fewer than 24 lines and see the header and footer bar containing some information. Finally, we were able to exit the program with Ctrl + Q.

In this section, we will add the following features to the text viewer:

Provide the ability to display files of any size.
Provide the ability for the user to scroll through the document using arrow keys.
Add cursor position coordinates to the footer bar.

Let’s begin by creating a new version of the code.

Copy the original code to a new file, as shown:

cp src/bin/text-viewer1.rs src/bin/text-viewer2.rs

This section is organized into three parts. First, we’ll implement the logic to respond to the following keystrokes from a user: up, down, left, right, and backspace. Next, we’ll implement the functionality to update the cursor position in internal data structures, and simultaneously update the cursor position onscreen. Lastly, we’ll allow scrolling through a multi-page document.

We’ll begin with handling user keystrokes.

Listening to keystrokes from the user

Let’s modify the run() method to act on user inputs and scroll through the document. We also want to record and display the current cursor position in the footer bar. The code is shown here:

src/bin/text-viewer2.rs

fn run(&mut self) {
    let mut stdout = stdout().into_raw_mode().unwrap();
    let stdin = stdin();
    for c in stdin.keys() {
        match c.unwrap() {
            Key::Ctrl('q') => {
                break;
            }
            Key::Left => {
                self.dec_x();
                self.show_document();
            }
            Key::Right => {
                self.inc_x();
                self.show_document();
            }
            Key::Up => {
                self.dec_y();
                self.show_document();
            }
            Key::Down => {
                self.inc_y();
                self.show_document();
            }
            Key::Backspace => {
                self.dec_x();
            }
            _ => {}
        }
        stdout.flush().unwrap();
    }
}

Lines in bold show the changes to the run() method from the earlier version. In this code, we are listening for up, down, left, right, and backspace keys. For any of these keypresses, we are incrementing the x or y coordinate appropriately using one of the following methods: inc_x(), inc_y(), dec_x(), or dec_y(). For example, if the right arrow is pressed, the x coordinate of the cursor position is incremented using the inc_x() method, and if the down arrow is pressed, only the y coordinate is incremented using the inc_y() method. The changes to coordinates are recorded in the internal data structure (the cur_pos field of the TextViewer struct). Also, the cursor is repositioned on the screen. All these are achieved by the inc_x(), inc_y(), dec_x(), and dec_y() methods.

After updating the cursor position, the screen is refreshed fully and repainted.

Let’s look at implementing the four methods to update cursor coordinates, and reposition the cursor on the screen.

Positioning the terminal cursor

Let’s write the code for the inc_x(), inc_y(), dec_x(), and dec_y() methods. These should be added as a part of the impl TextViewer block of code like the other methods:

src/bin/text-viewer2.rs

fn inc_x(&mut self) {
    if self.cur_pos.x < self.terminal_size.x {
        self.cur_pos.x += 1;
    }
    println!(
        "{}",
        termion::cursor::Goto(self.cur_pos.x as u16,
            self.cur_pos.y as u16)
    );
}
fn dec_x(&mut self) {
    if self.cur_pos.x > 1 {
        self.cur_pos.x -= 1;
    }
    println!(
        "{}",
        termion::cursor::Goto(self.cur_pos.x as u16,
            self.cur_pos.y as u16)
    );
}
fn inc_y(&mut self) {
     if self.cur_pos.y < self.doc_length {
         self.cur_pos.y += 1;
     }
     println!(
         "{}",
         termion::cursor::Goto(self.cur_pos.x as u16,
             self.cur_pos.y as u16)
     );
}
fn dec_y(&mut self) {
    if self.cur_pos.y > 1 {
        self.cur_pos.y -= 1;
    }
    println!(
        "{}",
        termion::cursor::Goto(self.cur_pos.x as u16,
            self.cur_pos.y as u16)
    );
}

The structure of all these four methods is similar and each performs only two steps:

Depending on the keypress, the corresponding coordinate (x or y) is incremented or decremented and recorded in the cur_pos internal variable.
The cursor is repositioned on the screen at the new coordinates.

We now have a mechanism to update the cursor coordinates whenever the user presses the up, down, left, right, or backspace keys. But that’s not enough. The cursor should be repositioned on the screen to the latest cursor coordinates. For this, we will have to update the show_document() method, which we will do in the next section.

Enabling scrolling on the terminal

We have so far implemented the code to listen for user keystrokes and reposition the cursor onscreen. Now, let’s turn our attention to another major issue in the code. If we load a document that has fewer lines than the terminal height, then the code works fine. But consider a situation where the terminal has the capacity to display 24 rows of characters, and there are 50 lines in the document to be displayed on text viewer. Our code cannot handle it. We’re going to fix it in this section.

To display more lines than is possible for the screen size, it is not enough to reposition the cursor. We will have to repaint the screen to fit a portion of the document in the terminal screen depending on the cursor location. Let’s see the modifications needed to the show_document() method to enable scrolling. Look for the following lines of code in the show_document() method:

for line in 0..self.doc_length {
      println!("{}\r", self.doc.lines[line as
          usize]);
      }

Replace the preceding with the following code:

src/bin/text-viewer2.rs

if self.doc_length < self.terminal_size.y {                     <1>       
    for line in 0..self.doc_length {
        println!("{}\r", self.doc.lines[line as
            usize]);
    }
} else {

    if pos.y <= self.terminal_size.y {                          <2>    
        for line in 0..self.terminal_size.y - 3 {
            println!("{}\r", self.doc.lines[line as
                usize]);
        }
    } else {
        for line in pos.y - (self.terminal_size.y –
            3)..pos.y {
            println!("{}\r", self.doc.lines[line as
                usize]);
        }
    }
}

The code annotations in the show_document() method snippet are described here:

First, check whether the number of lines in the input document is less than the terminal height. If so, display all lines from the input document on the terminal screen.
If the number of lines in the input document is greater than the terminal height, we have to display the document in parts. Initially, the first set of lines from the document are displayed onscreen corresponding to the number of rows that will fit into the terminal height. For example, if we allocate 21 lines to the text display area, then as long as the cursor is within these lines, the original set of lines is displayed. If the user scrolls down further, then the next set of lines is displayed onscreen.

Let’s run the program with the following:

cargo run –-bin text-viewer2

You can try two kinds of file inputs:

A file where the number of lines is less than the terminal height
A file where the number of lines is more than the terminal height

You can use the up, down, left, and right arrows to scroll through the document and see the contents. You will also see the current cursor position (both x and y coordinates) displayed on the footer bar. Type Ctrl + Q to exit.

This concludes the text viewer project. You have built a functional text viewer that can display files of any size, and can scroll through its contents using the arrow keys. You can also view the current position of the cursor along with the filename and number of lines in the footer bar.

Note on the text viewer

Note that what we have implemented is a mini version of a text viewer in under 200 lines of code. While it demonstrates the key functionality, additional features and edge cases can be implemented by you to enhance the application and improve its usability. Furthermore, this viewer can also be converted into a full-fledged text editor. These are left to you, the reader, as an exercise.

We’ve completed the implementation of the text viewer project in this section. The text viewer is a classic command-line application, lacking a graphical user interface (GUI) where mouse inputs are needed. But it is important to learn how to handle mouse events, for developing GUI-based terminal interfaces. We’ll learn how to do that in the next section.

Processing mouse inputs

Like keyboard events, the Termion crate also supports the ability to listen for mouse events, track the mouse cursor location, and react to it in code. Let’s see how to do this here.

Create a new source file called mouse-events.rs under src/bin.

Here is the code logic:

Import the needed modules.
Enable mouse support in the terminal.
Clear the screen.
Create an iterator over incoming events.
Listen to mouse presses, release and hold events, and display the mouse cursor location on the terminal screen.

The code is explained in snippets corresponding to each of these points.

Let’s first look at module imports:

We’re importing the termion crate modules for switching to raw mode, detecting the cursor position, and listening to mouse events:

use std::io::{self, Write};
use termion::cursor::{self, DetectCursorPos};
use termion::event::*;
use termion::input::{MouseTerminal, TermRead};
use termion::raw::IntoRawMode;

In the main() function, let’s enable mouse support as shown:

fn main() {
    let stdin = io::stdin();
    let mut stdout = MouseTerminal::from(io::stdout().
        into_raw_mode().unwrap());
    // ...Other code not shown
}

To ensure that previous text on the terminal screen does not interfere with this program, let’s clear the screen, as shown here:

writeln!(
    stdout,
    "{}{} Type q to exit.",
    termion::clear::All,
    termion::cursor::Goto(1, 1)
)
.unwrap();

2. Next, let’s create an iterator over incoming events and listen to mouse events. Display the location of the mouse cursor on the terminal:

for c in stdin.events() {
    let evt = c.unwrap();
    match evt {
        Event::Key(Key::Char('q')) => break,
        Event::Mouse(m) => match m {
            MouseEvent::Press(_, a, b) |
                MouseEvent::Release(a, b) |
                MouseEvent::Hold(a, b) => {
                    write!(stdout, "{}",
                    cursor::Goto(a, b))
                    .unwrap();
                    let (x, y) = stdout.cursor_pos
                        ().unwrap();
                    write!(
                        stdout,
                        "{}{}Cursor is at:
                        ({},{}){}",
                        cursor::Goto(5, 5),
                        termion::clear::
                            UntilNewline,
                        x,
                        y,
                        cursor::Goto(a, b)
                    )
                    .unwrap();
            }
        },
        _ => {}
    }
    
    stdout.flush().unwrap();
}

In the code shown, we are listening to both keyboard events and mouse events. In keyboard events, we are specifically looking for the Q key, which exits the program. We are also listening to mouse events — press, release, and hold. In this case, we position the cursor at the specified coordinates and also print out the coordinates to the terminal screen.

3. Run the program with the following command:

cargo run --bin mouse-events

4. Click around the screen with the mouse, and you will see the cursor position coordinates displayed on the terminal screen. Press q to exit.

With that, we conclude the section on working with mouse events on the terminal, and also this introduction to terminal I/O management using Rust.

Summary

In this article, we learned the basics of terminal management by writing a mini text viewer. We saw how to use the Termion library to get the terminal size, set the foreground and background colors, and set styles. After this, we looked at how to work with cursors on the terminal, including clearing the screen, positioning the cursor at a particular set of coordinates, and keeping track of the current cursor position.

You’ve learned how to listen to user inputs and track the keyboard arrow keys for scrolling operations, including left, right, up, and down. We wrote code to display document contents dynamically as the user scrolls through it, keeping the constraints of the terminal size in mind. As an exercise, you could refine the text viewer, and also add functionality to convert the text viewer into a full-fledged editor.

Learning about these features is important for writing applications such as terminal-based games, editing and viewing applications and terminal graphical interfaces, and for developing terminal-based dashboards.

To read more about building fast and secure software for Linux/UNIX systems, check out Prabhu Eshwarla’s book Practical System Programming for Rust Developers. Explore Rust features, data structures, libraries, and toolchain to build modern systems software with the help of hands-on examples.

Upgrading Existing .NET Apps to .NET 5

Packt — Fri, 30 Apr 2021 12:53:00 GMT

Upgrading Existing .NET Apps to .NET 5

Upgrading an app to its latest platform version, just for the sake of upgrading, is usually not a fun exercise, especially if there is no new and exciting feature to be added. Fortunately, Microsoft has provided solid inter-compatibility between libraries developed for earlier versions of .NET and .NET 5 to make the migration process much smoother. The cherry on top is the underlying performance enhancement that will make the existing code run faster without any significant transformation by the app developer.

There are some technologies that Microsoft has decided not to port to the .NET 5 platform, such as Windows Communication Foundation (WCF) Server, Windows Workflow, and ASP.NET web forms. We will discuss the options to replace these technologies where it is feasible to do so.

In this article, you will learn about the following topics:

Choosing the migration approach
Points to ponder regarding an as-is upgrade

We’ll assume that you are already familiar with .NET Framework and Entity Framework as we are upgrading these to the latest platform. We will focus on the core aspects of these technologies that require transformation during migration to .NET 5.

Choosing the migration approach

Before starting the app migration process there are some big decisions that need to be made upfront.

If you are upgrading from any previous version of .NET Core, the implications are less severe simply because .NET 5 is a continuation of the same platform and the upgrade process is much simpler.

Migrating from .NET Framework, however, requires a bit more thought. There are a number of factors that will impact the approach, including the following:

The size and footprint of the solution
The number of different technologies that comprise the solution (for example, desktop apps, web apps, and REST APIs)
Whether an unsupported technology (such as WCF Server) or .NET runtime feature (such as .NET Remoting) is used

These factors can influence the .NET migration approach and in particular the decision about whether to migrate the whole solution at once or try to do it in different stages. Let’s review the pros and cons of both approaches.

Big bang approach

For small- and medium-sized solutions, it makes sense to upgrade the entire solution stack at once from .NET Framework to .NET 5. For large solutions, it might not be too practical due to time constraints. You would need to maintain two separate codebases, one each for .NET Framework and .NET 5, during the migration process, and if any new feature needs to be implemented, it could become hard to keep the two code bases in sync.

The benefit of this approach is its simplicity. Instead of worrying about compatibility issues between intermediate phases of the upgrade, we just have to ensure that the final version of all the solution pieces is compatible with .NET 5.

Gradual upgrade

An alternative approach is to identify the least-dependent libraries first. These are the projects that do not depend on other projects in the codebase. A common example of such projects is a library containing model definitions or shared helper functions. Once these libraries have been upgraded to .NET 5, we look for the next set of libraries that are only dependent on previously upgraded components. We then gradually move up the dependency ladder to migrate the components with all their dependencies previously migrated.

The .NET team has made it easier by making it possible to multi-target an assembly for multiple platforms. So, you can write code once and target it for both .NET Framework and .NET 5 platforms. This will obviously work for scenarios where feature sets across both platforms are the same, or similar. Once the shared libraries have been migrated, the executing app (desktop or web app) can be migrated to the latest platform to take advantage of .NET 5’s specific features. The complexity arises with this approach when a particular API is not supported on .NET 5. We would have to write a conditional compilation code so that unsupported API is only used when code is compiled for the .NET Framework platform.

Once the choice between the gradual versus big bang approach has been made, the next decision is whether to introduce any changes in the solution architecture.

Lift and shift versus architectural transformation

In the lift and shift approach, we try to minimize any fundamental changes to code and its constituent components. Once the application is up and running on the latest platform, architectural changes are taken aboard as a separate project. This is a simpler approach and there is less risk of upgrade failure, but the new platform must support the components from the previous platform.

For example, if you have built a WCF Server instance or have implemented inter-process communication using .NET Remoting, it is simply not possible to upgrade these technologies on an as-is basis on .NET 5 as .NET 5 does not support them.

Architectural transformation makes sense when new features are expected to be implemented as part of an upgrade exercise or in the near future. It is best to leverage the latest platform features and technologies to reduce the technical debt on an older platform.

The final decision that we need to make is about whether and how we upgrade third-party libraries.

Replacing third-party libraries with platform-provided features

A similar concern to the architectural changes that are discussed in the previous section is about making code improvements. This is especially tempting for third-party libraries when a newer alternative is provided by the framework itself.

In .NET Core and .NET 5, Microsoft provides a system API for JSON parsing and serialization named System.Text.Json. Similarly, libraries for dependency injection (DI), Microsoft.Extensions.DependencyInjection, and logging, Microsoft.Extensions.Logging, eliminate the need to use corresponding third-party libraries, especially for basic use cases. If you are using advanced features of existing libraries, then it is best to consult the documentation of the newer alternative to understand whether every feature is available as a drop-in replacement or whether significant development is required.

Also, sometimes it takes less effort to transform the feature than to keep trying to use the existing code or library in a backward-compatible fashion. For example, in .NET 5, it is easier to read application configuration from JSON files instead of *.config XML files, so it may be worthwhile upgrading the configuration reading code unless it is significantly customized in the existing codebase.

We will now review the points that would generally be applicable to most .NET like-for-like migration projects.

Points to ponder regarding an as-is upgrade

A transformation project can easily get quite large and complex, but sometimes it is possible to score some worthwhile technical improvements in the solution, where comparatively little effort brings good rewards in terms of code maintenance and performance.

With that in mind, the following subsections make some general points that will be applicable to most upgrade projects.

Unsupported technologies

While most of the .NET Framework technologies can be migrated relatively easily to the .NET 5 platform, some features and technologies do not have a direct replacement and will require redesign or, at a minimum, replacing code from a different library. In addition, due to the fast-paced nature of .NET Core/.NET 5 development, many features in libraries such as Entity Framework Core or ASP.NET Core get marked as obsolete or deprecated in newer versions. So, it is best to review release notes when migrating to newer versions of these libraries.

In this section, we will review what major .NET Framework features are not available and the possible approaches to replace these features.

WCF Server

WCF was originally released in November 2006. Since then, it has been a very popular inter-service communication mechanism for .NET developers. It has vastly improved technology compared to ASMX web services. Its major strength is the support for various communication protocols, inter-server communication, and intra-server communication with minimal coding changes.

While still used in many legacy applications, Microsoft has decided to not support WCF Server on the .NET Core/.NET 5 platform in favor of modern technologies such as REST APIs over HTTP or Google RPC (commonly known as gRPC) for contract-based RPCs. Both alternatives are cross-platform and support the most popular programming languages and cloud platforms.

Tip

If you still want to continue development with WCF Server, one option is to consider the CoreWCF project, which is an open-source port of WCF to the .NET Core platform: https://github.com/CoreWCF/CoreWCF11.

At the time of writing, this project is still under development and not production-ready.

The recommended approach is to use the REST APIs or gRPC for new features. Alternatively, you can upgrade the WCF client to .NET 5 and continue to use the WCF Server on .NET Framework until you have the time resources available to do the migration.

ASP.NET Web Forms

ASP.NET Web Forms is another technology that is being axed in .NET 5. Microsoft recommends migrating to Blazor WebAssembly for modern web UX development. Blazor supports C# and is quite scalable. The transformation from Web Forms to Blazor is not a simple task and requires considerable planning.

The following Microsoft article has guidance on transforming ASP.NET Web Forms apps to Blazor WebAssembly: https://docs.microsoft.com/en-us/dotnet/architecture/blazor-for-web-forms-developers/migration.

ASP.NET Web Forms was introduced in 2002 and now (if not before) is a good time to port the code to newer technologies. If you could consider using a non-C# technology, modern JavaScript-based UI frameworks such as Angular and React provide a robust feature set to build a sleek UX.

Windows Workflow Foundation (WWF or WF)

WWF’s primary feature set is to reduce coding for developing and automating repeatable business processes. It does so by providing an API, an in-process workflow engine that supports long-running processes, and a visual designer.

Compared to WCF Server and Web Forms, WWF’s market footprint is somewhat reduced. There is no direct alternative to this technology on .NET 5, but most of its functionality can be redesigned using a combination of gRPC, REST APIs, Azure Logic Apps, and Power Automate.

Like CoreWCF, there is an open-source fork of WWF available that is still in the experimental phase: https://github.com/UiPath-Open/corewf.

We will now review some Windows platform-specific features that cannot be used from code when targeting .NET 5.

Unsupported Windows OS-specific features

Aside from the technologies mentioned in the preceding section, some platform-specific features for Windows OS are not available in .NET 5 platform libraries. These features should generally be of concern to Windows desktop developers when migrating to .NET 5.

AppDomains

AppDomains were primarily used to isolate application processes from each other. For modern applications, a better alternative for process isolation is to use containers, which are more versatile and flexible. There is also the AssemblyLoadContext class, which supports dynamically loading assemblies at runtime.

For ease of migration, .NET 5 provides some of AppDomains’ APIs; however, not all methods are guaranteed to behave in the same way. Some will throw PlatformNotSupportedException, some will do nothing, and others will continue to function as per previous behavior.

Remoting

A related feature to AppDomains is remoting, where processes living in different app domains can communicate with each other. This is no longer supported.

For communications across the network, REST and gRPC are recommended. For interprocess communication on the same machine, MemoryMappedFile or System.IO.Pipes can be used.

Code Access Security (CAS)

Sandboxing applications on the Windows platform allowed applications to execute in a constrained environment with restricted access to resources. This feature has been retired from .NET Framework itself, and so has not been ported to .NET Core or .NET 5.

The cross-platform alternative is to restrict process privileges by using containers and/or constrained user accounts.

Security transparency

Security transparency was used in sandboxed applications to declaratively specify access privileges required by code. This is generally used in Silverlight applications. As CAS is not supported, the related feature of security transparency is also not supported. The alternative is also the same as CAS, which is to use virtualization, containers, and user accounts to control the process privileges.

Transforming application configuration

In .NET Framework, application configuration is typically specified in app.config or web.config files in XML format.

An example appSettings section appears as follows:

.NET Framework implemented a hierarchy of configuration files, where a parent file’s settings can be inherited or overridden by a more specific file. For example, machine-specific settings can be specified in machine.config and then a web application can inherit or override these settings for its process in a web.config file.

These settings are accessed by developers using the System.Configuration namespace. This namespace is still available in the .NET Core world, but a more flexible approach with usually minor code changes is to use the Microsoft.Extensions.Configuration namespace, which offers considerable improvements.

The benefits of using the Microsoft.Extensions.Configuration namespace includes:

Support for hierarchical plain old CLR object (POCO)-based settings
Built-in binding and validation for non-string values, such as int, bool, and decimal
Uniform handling and choices of configuration store, such as JSON files, XML files, in-memory objects, command-line arguments, and environment variables
Support for encrypted and cloud-specific stores, such as Azure Key Vault or Azure App Configuration

Choosing the Entity Framework version

The majority of applications need to access databases for one use case or another. Entity Framework is a very popular choice for .NET developers as an object-relation mapping (ORM) tool to access the data layer in .NET Code.

The most recent major version of Entity Framework is Entity Framework 6. It is fully supported by Microsoft on the .NET Core platform, but all new development should be done on Entity Framework Core. EF Core supports .NET Framework (up to EF Core v3.1) and .NET 5 (all EF Core versions).

EF Core and Entity Framework do not have 100% feature parity; there are many new features in EF Core and some that will never be ported from Entity Framework to EF Core (due to low usage or radical design change), while some features are slated to be ported in future EF Core versions.

Refer to the official documentation for a detailed overview of the features that are not available on either platform: https://docs.microsoft.com/en-us/ef/efcore-and-ef6/.

Using .NET Framework compatibility

A common migration approach is to port your libraries to .NET Standard first. .NET Standard libraries can be referenced from both .NET Framework and .NET 5 assemblies, which works great as an interim solution until both frameworks are in use in the organization.

With .NET Framework compatibility mode, .NET Standard projects can make references to .NET Framework libraries as if they were compiled for the .NET Standard platform. This helps remove a major migration blocker where migrating dependent libraries to newer platforms is not feasible in a short time frame.

Of course, having a compatibility layer does not automatically guarantee that all .NET Framework APIs will work on the .NET 5 platform. The APIs that are not supported will either throw PlatformNotSupportedException or might do nothing in some cases.

The vast majority of APIs are now ported on the .NET 5 platform, so using .NET Standard is a useful way to reference third-party NuGet packages that have not been updated by their developers despite having full API available on the new platform.

During our example migration exercise, we will learn about using the .NET Portability Analyzer tool, which can detect and inform about potential API incompatibility on the new platform.

Upgrading third-party NuGet packages

When using third-party NuGet packages, you can use one of the following approaches:

If the current NuGet package version is .NET 5- or .NET Standard 2.1-compatible, then use that version.
If the current version does not support .NET 5/.NET Standard 2.1, but the later NuGet package version does, it’s likely a good candidate to be used. We need to review any breaking changes in the new version to assess the code refactoring effort required to use the new version.
The vast majority of active and popular NuGet packages fall under the first two cases. However, if there is no compatible version, then it is possible to use the .NET Framework library from .NET Standard code using the compatibility mode as described in the preceding section.
Another option is to use an alternative library, but this could potentially take significant time and cost to implement.

We have now covered enough theoretical ground to understand the considerations and pitfalls of migration.

Summary

In this article, we’ve learned about different migration approaches and the pros and cons of each. There is no one-size-fits-all solution. The preferred approach depends on the compatibility of the technologies involved, the size of the existing code base, and the extent of dependence on the third-party NuGet libraries.

Sometimes it’s not possible to use the legacy technology, as we saw in the case of WCF Server, WWF, and Web Forms. Other technologies, such as gRPC, Blazor, and REST, can usually be used to replace the existing solutions in such cases.

Discover practical examples of upgrading .NET apps to .NET 5 and get ready to make the most of .NET 5’s features with Hammad Arif and Habib Qureshi’s book Adopting .NET 5.

Exploring SharePoint Migration

Packt — Tue, 27 Apr 2021 08:18:59 GMT

Migrating data is a tough role in any organization. Getting all the data from source to destination is a challenging task, especially with Microsoft SharePoint, mapped drives, and other storage resources. Users will complain about data being missing, even if it is irrelevant. In some cases, they will be irritated or even combative, because they are afraid of change. This overview will give you valuable insights to help you master these challenges to deliver a painless migration.

Getting ready: know your content

When contemplating a SharePoint migration, the first piece of advice is to plan and make sure you understand your users’ data, some of their processes, and the content they are using. You will also need to find out what data is relevant and what data may not be relevant before moving to the new environment. You will find this to be a daunting task when dealing with mapped drives because people do not know what the latest versions of files are in most cases; for example, some may be named FinalFinal. To make sure you’re using the correct files, you must work with the customer to clean up any irrelevant files before going forward. You have lots of choices when it comes to storage locations due to SharePoint and OneDrive being available. Don’t go back to mapped drives if you can avoid it, but if you do, clean up those files before moving them.

When looking at content, you must be prepared to classify and reclassify files as needed and tag them within the new environment correctly. The use of managed metadata and/or columns to describe the content is key for search results. Securing the data based on those classifications comes into play so that once they’ve been migrated to a document library, they can be secured. Files within the libraries can also be secured individually if needed. Breaking inheritance is also a concern at this point when you’re talking to your customer because you do not want to do this a lot when you’re setting up content. So, be mindful of how you set up new libraries in the new environment.

Here is a list of areas to pay attention to before migration:

File size
Individual permissions
URLs (file paths) and file names
File sizes
Character limitations
Custom solutions
Branding
InfoPath
Workflow state and history
Permissions (do you have access to all the files?)
Folders with more than 5000 items
Unsupported site templates
Orphaned users
Checked out files
Unsupported list templates
File extensions

We must also remember that even with mapped drives, we can also migrate to OneDrive. Security and various options for folder organization will help the customer manage those files. It’s not like SharePoint, but if a customer wants to implement this type of storage location, be ready to answer questions. This is contingent on you being in a hybrid environment where you already have Microsoft 365 set up.

If you’re working with MS SharePoint, you will also need to look at versioning settings, column types, and any updated aspects that are moving from one version of SharePoint to another. Help them understand that moving to this new location will give them more control over the data from a security aspect; waiting for an admin to update security for new and existing users within mapped drives is also possible.

This sometimes gets them excited about moving and helps them relax, which will make your users feel as though everything is going to be OK. Show them that you have investigated everything and build confidence in the migration by having meetings about this migration. You can even suggest the use of retention libraries for holding documents online either with OneDrive or SharePoint so that the customer understands there are choices when it comes to how old documents can be handled. Processes can be put in place for handling new files that need to be moved to a separate location for safekeeping as well.

As we mentioned previously, this is not an easy task. You are in the hot seat to figure out how to get 100% of your identified content from one version to the other. We will talk about some of the areas to focus on and which migration method will work for your migration scenario. We have also identified some migration tools within the method to give you a breakdown of my experiences with those tools.

We will also talk about creating schedules for your departments and testing with users as we also want to break down the use of preliminary migrations and final migration testing. If you have not, and we will say this again, create new dev and test environments to make sure you can provide areas within the environment where users can test their content and have new ways of accessing it.

As part of our environment, we should have dev, test, and production environments, which gives us a complete, functional SharePoint environment. This means we have a place we can test out migrations and have the user test them too. The last thing you want to do is a test in your test and production environments; always do this during the development phase. If you have these environments set up, then we can move on and find out what we need to do in order to start moving content from our old farm to the new one. Make sure all your farms have been configured correctly and are the same so that the results of the tests will be the same across the board. Configuration management is important!

During migrations, you also want to make sure you put source content in read-only mode when doing final migrations so that nothing is changed during this process. There are many things you need to make sure are in place before we start this migration. Take full backups of content before the migration, as well as once you have got the content over to the new farm. This backup should be taken once you have settled all issues and confirmed that all the errors have been fixed with the customer. This helps create a baseline that acts as a safeguard from the migration of both the source and target farms.

Know your customers

Identify all the departments that are being migrated and create a list of departments and users (if possible). If you have departments that are ready to move and eager to take advantage of new technology, then work with those customers first. These departments are the ones that will be successful and will give you a chance to build a good story going forward. Make sure to interview all departments before picking and choosing a schedule as well.

The following are some example questions you should ask when performing department interviews:

What are you currently using SharePoint for?
How big are some of the larger files that you will be migrating?
Is there anything you want to change on your current site?
Do you have any automated processes?
Do you have any custom applications?
Are there any file shares that will be migrated?
What do you do day-to-day with SharePoint?
Do you have any staff that can be contacted and can help drive the migration process?
When would you like to move?
Do you have any immediate or future goals for the product?

It is crucial that the migration is successful. When you fail in front of IT staff, it is OK; you can recover since it’s a small subset of the project. However, when you get to the migration stage, this affects the company because everyone is watching. This also brings a spotlight on why some users do not like SharePoint. Do not give them any ammunition to criticize SharePoint because it can cause issues and it’s something you will have to live with for a while. Believe us, success is the goal here.

Some of the things you can do to be more in tune with your customers or department include understanding what they do. Knowing what they do and how they use SharePoint can go a long way in supporting the department during the migration process. If possible and if you have time, find out who used to work in those departments that might have data that is still valid in the source content. Knowing some of those user accounts that may not be in Active Directory now will help to troubleshoot errors going forward. Some tools that are used when migrating lists and libraries do not work well when versioning items that could have a user account specified in a people and group column because, when the item is migrated, it will fail. Use the tool mapping component to map users and groups if needed. By doing this, accounts that are not there can be masked with the admin account or another account so that the item does not fail during the migration.

Find out what your customer or department needs from you. Whether it is hand-holding or more information for them to feel comfortable, make sure you provide it. There might be multiple questions you wish to ask, including:

Are you migrating a file share or SharePoint site?
Do you have access to the file share or SharePoint site?
How large are the files that are being migrated?

Reduce the likelihood of any scheduling hold-ups by interviewing all departments before you finalize your schedule. If there is a person on the staff that would like to help or be that go-to person, find that out now so that they can be your liaison with the rest of the team.

Find out the immediate or future goals of the department. This is crucial so that you can get an understanding of where they would like to go with SharePoint. This also gives you the opportunity to discuss how SharePoint can help them manage data more efficiently. Having a demo ready for some of the new features within SharePoint can show you are prepared and that these features are working as they should in the new version. This gives the business analyst some information on what could be done to help them in the future. This is the time to start being proactive and not reactive with your customers. Start fresh and make this transition useful for everyone.

Migration — choosing your weapon of choice

When choosing how we migrate, we must look at the content and versions of SharePoint we are working with. Some questions that you should be asking yourself are as follows:

What method should I use to get this task completed?
Should I purchase a tool to perform migration?
What are some other methods we can use to migrate content from the source to the destination?

There are a few ways to migrate content in SharePoint:

Content database migration
Using a migration tool
PowerShell migration
Manually using Windows Explorer

Let’s take a look at these in more detail.

Content database migration

This is the out-of-the-box task of moving content from one farm to another. This supports migration from version to version and cross-web applications for on-premises environments. There are technical points that need to be addressed as part of both migration paths when using this method. This provides a better capture of the site from a bigger standpoint, but it does not give you some of the capabilities you may need in order to clean up content beforehand. This is best used when custom solutions are not involved, and the content is mostly out of the box.

Using a migration tool

This depends on whether you’re using an external application to move content from one SharePoint site to another site. These tools can be used for either on-premises or cloud migrations. This also means we can migrate from one version of a SharePoint farm to another SharePoint farm. Using a migration tool provides us with flexibility for scheduling, picking specific content, using mapping capabilities to map columns in lists, and updating permissions on items that may be held by users who no longer work at the company, to name a few. Governance tools also come into play with tools you select, as many companies provide those to help you understand your environment before migration.

One thing we do to speed up the migration process is to create as many server-named sites as possible to extend our web applications. This will give you many URL points of entry for content migrations so that you can use separate servers to push the content through, instead of using a DNS name where all traffic will hit one or two servers. The server resources are then separate from one another and you can have different admins pointing different tool installations to different servers. You will see no bottlenecks pushing content through as long as your SQL server can keep up with the influx of data.

Tools also allow you to make choices for other Microsoft cloud services and third-party solutions such as Google, Dropbox, and OneDrive. If you have a tool that can connect to those types of services and you want to migrate content from those repositories, this is the way to do it. A migration tool provides such integrations and can be helpful in moving these documents over to a new platform. It will also leave behind database errors found in the content, such as errors with solutions that were left behind from other versions, plus other content that may have faced errors due to database issues.

You can also use the tool in combination with content database migration and migrate your databases first, before using the tool on the weekend to push incremental changes to the content. This cuts down how long it will take you to move content and save you money since with most tools you must pay per GB. Unfortunately, you cannot use this method when moving to the cloud.

Important note

Remember: your logging files will grow while you’re completing this process, and large amounts of logging data will be created on your SQL and SharePoint servers. Make sure that all your logging services for IIS and SharePoint are pointed to the Data or Log drive on your machine and that it has a lot of space during this process!

PowerShell migration

PowerShell migration is another way we can migrate content. It is more tedious and requires you to be skilled in PowerShell. We do not recommend this method for newbies or admins who do not have this skill set already. The chances of errors occurring are high, and you do not want that when you are on a schedule. It is not the time to practice PowerShell at this point.

Tip

Some examples of PowerShell being used as a migration tool can be found on Microsoft’s website at https://docs.microsoft.com/en-us/sharepointmigration/overview-spmt-ps-cmdlets.

Manual migrations

As an example, let’s say we have a folder on a server that is being used as a shared folder. This folder contains documents that need to be migrated to a SharePoint library. We can take these files from the shared folder and copy them directly into the library using the Windows Explorer functionality within the SharePoint library. This is an easy way to migrate documents from desktops, but the limitation is that you can only copy up to 100 documents at a time. You also do not have the opportunity to add column information or metadata to these documents as they are being uploaded.

Important Note

Beware! Do not make the mistake of having a live site on a destination farm where you have not finished configuring your services. Make sure you have completed all configurations before you put any new ‘live’ sites in your new production environment. If you need to work with some of the departments and have sites that are live while migrating, ensure your dev and test environments are set up so that you can test any migrated content before it’s put into production as a live site. You can then make configuration changes there and not in production to test the farm.

There are other ways we can use manual processes to move data as well. Microsoft Office tools such as Microsoft Access and Microsoft Excel allow us to move content from one list to another or one farm to another. All lists are basically Excel and Access databases.

So, capturing columns and pulling data back into another list is fairly straightforward using PowerShell and even just Access itself. Look around and see what you can find. In some instances, this may work when you are just moving small items or concentrating on one document library and don’t want to purchase something, or you don’t want users bothering the admin when it comes to moving the list through Central Administration.

The Central Administration tool, which can be used to export lists and libraries, can be seen in the following screenshot:

Figure 1 - Exporting a site or list

As you can see, there are many ways to migrate content from SharePoint. Using the tools that work for your situation is the name of the game. Do not just take someone else’s word for it. Some tools work better for some people than others. It’s all about what you want to achieve, not how expensive or popular a solution is. Even within SharePoint, we can pull content from top-level areas and import them. In some cases, this would not work, while in others, it could be the best solution to use. Just make sure it works for you and your requirements.

Now that we have talked about the different ways we can perform migrations, next we will learn about the types of migrations that can be achieved.

Types of migration

In this section, we will cover different types of migration. This will provide you with a use case to follow once you start the migration process. The types of migration that can be achieved are as follows:

Version to version
Server to server
Service to service
Cross-web applications
Cloud migrations from on-premises
Third-party to on-premises and cloud
Files shares

Version to version: A version-to-version migration is when you migrate from a SharePoint 2013 farm to a SharePoint 2019 farm. However, as part of that migration, you need to validate your content through the in-between version of SharePoint — in this case, SharePoint 2016. This also needs to be a content database migration method.

Server to server: Server-to-server migrations can be easy or may become very complex, depending on the way the content was structured in SharePoint. Server-to-server migrations could be when you migrate from one cloud to another cloud where you have the same version of SharePoint, or it could be when you need to upgrade the server platform from Windows Server 2012 R2 to Windows 2016 Server. This could be for licensing or supportability reasons, or just to keep your servers up to date with the latest and greatest server technology. If you are migrating from SharePoint 2007 to SharePoint 2019 using the content database migration method, this could be considered a server-to-server migration as well because when lifting content from 2007, you have to take it through the different versions of SharePoint to get the content to 2019.

When moving through 2010, 2013, and 2016, each server would have to be set up and configured to support the migration. There is also a step in the 2010 migration that will upgrade your workflows and InfoPath forms during the process, and there’s a tool to migrate you to 2019 from that point. You would have to manually migrate the business data catalog (BCS in 2019) from 2007 to 2010, for which you can refer to https://docs.microsoft.com/en-us/previous-versions/office/developer/sharepoint-2010/ff817559(v=office.14). After that, the services database could be moved through this process as well to keep those services intact as you move through the migration. You will also need to migrate classic users to claims, which occurs during the migration from 2007 to 2010. For more information on migrating from classic-mode to claims-based authentication in SharePoint 2013, refer to https://docs.microsoft.com/en-us/sharepoint/upgrade-and-update/migrate-from-classic-mode-to-claims-based-authentication-in-sharepoint-2013.

Service to service: These migrations can be complex. We were recently on a migration from Rackspace to AWS and it was challenging due to the size of the content databases that we had to work with. Getting such databases transferred to a new service is tough, especially when you have small maintenance windows. The one thing to watch out for is the size, but also what type of web application configuration you choose.

Say, for instance, you chose Path-Based Web Applications as your standard application. Now, in terms of migration, all content needs to be migrated all at once, regardless of how many databases you have in that web application. If you do not move all the content databases at once, your users will have no way of getting to the content because there is no way to split the URL so that it talks to two separate farms.

However, you can migrate Host Named Site Collections one at a time or all at once. This is because the Host Name is the key to getting to the content, and it is associated with the site collection and not the web application. This is great when it comes to these types of situations, where content can be very hard to move to another service or farm. Plan accordingly so that you do not get caught in a situation like this, especially when you’re managing the sizes of your content databases.

Cross-web applications (internal farm): These migrations also happen during the production use of a SharePoint farm. You need to understand how or why you need to use any other migration methods mentioned at the beginning of this section. The approach you should use for list and library migrations would be to use a tool or use the area within Central Administration to extract the site, list, or library and then move the content from one site to another. Some tweaks may be involved here, but this is something you should explore, depending on your requirements.

Migration to the cloud: This can only be achieved using a tool or a very time-consuming manual method. Of course, a tool is preferred in this scenario due to the amount of time you would have to spend going through files and uploading them to SharePoint Online. The manual process would be to download all files to a local system and then upload them back into the new farm libraries. This would mean reapplying all permissions, as well as losing version history. We can see why a tool is very important when migrating to the cloud. Most tools charge you per GB, so please budget for this!

We remember migrating from DocuShare a few years ago and the only tool available was the command line. During that process, you could export all the files, but a lot of information was lost. At the time of writing, there are lots of command-line tools, such as JSOM, that can help define these files and permissions. They are exported in a flat-file structure to help pull that information into SharePoint and apply some metadata from the old system, which in some cases is not SharePoint.

You can also migrate many different types of information using tools to SharePoint or OneDrive. Some tools give you different capabilities with different applications. Some connect to Google Docs, Dropbox, and even some other applications. We have seen some tools, such as Saketa, offer Microsoft Teams migrations as part of their toolset.

Microsoft also has a free tool that can be used to perform migrations from an on-premises environment to the cloud. We have only used this tool once and it works as it should. However, at the time, it didn’t have a lot of bells and whistles compared to what other tools offer. We believe they have made updates to this tool, so if you are looking for one, try it and see if it will work for your project.

Important note

Microsoft has its own migration tool that only migrates to the cloud. You can find more information at https://docs.microsoft.com/en-us/sharepointmigration/introducing-the-sharepoint-migration-tool.

File shares: File shares are always forgotten, but the return on investment in migrating these files to SharePoint is huge. When we leave these files shared on servers, you are requiring these files to be hosted by one or many servers with extended storage space. These servers cost you money to run and always require disk space upgrades. Instead, pull these files into SharePoint and use the databases for sharing and managing these files.

OneDrive can also be used to hold personal files and get rid of MySites and the storage comes for free with your plan. The SharePoint 2019 Server must be set up to use a hybrid connection to the cloud.

The users win as well because they gain functionality when using SharePoint, and your help desk will love you for it. With this, you have the benefit of versioning and much more control over your files. You can also place the files into segregated site collections/document libraries so that there will be no slips in data security, unlike file shares. They will also not be calling the help desk to add users to certain folders, which will cut down administration use as well. Users can also use files for workflow processes, versioning, and other collaborative means. They can include these files in links throughout sites in order to share information using the latest version of the file via Manage Copies.

Migration tools

Most tools run on any OS platform and provide a lot of options when it comes to migrating using mappings, PowerShell, and scheduling. Our experience with ShareGate has been great, but it does not offer an array of solutions as it seems to concentrate on the migration and governance space. As far as the tool goes, it is great and provides everything you need to prepare for a smooth migration process. We have not had many issues using the product, and we would say that it is first class!

ShareGate has been the tool of choice for our migrations. We really like both ShareGate and Saketa. Unlike other tools in this space, ShareGate gives you a great desktop and many functionalities you will not see in other tools. The tool is inexpensive and runs on any Windows platform. Again, pricing is tiered, so you can find a comfortable spot that fits your budget based on your data requirements.

Important note

If you are performing an on-premises migration, make sure to try out content database migrations first. This will save you money in the long run. You can then use ShareGate to push the deltas needed to finalize the updates to the user’s content when performing on-premises migrations.

Some of the features of ShareGate are as follows:

Auto-generates PowerShell scripts for all types of migrations with copy options
You can pre-test your migration to figure out errors before performing a real-time migration
Governance reporting features for pinpointing issues with governance
Gives you the option to import user and group mappings from Excel files
ShareGate Shell can be used to migrate on schedules and allows you to use PowerShell within the tool
Advanced options for managing more complex migration strategies
Provides a connection manager for performing SharePoint and OneDrive migrations
Performance improves when using Insane Mode for migrations
General bug fixes and updates are provided regularly
Easy to use.

The following is a list of some of the tools that can be used for SharePoint migration:

Figure 2 - Tools that can be used for SharePoint migration

AvePoint also has a new migration tool called Fly that does not require any server resources. This really changed the game for the company’s migration offering from our perspective. It can be installed free of charge from the AvePoints enterprise application interface and is similar to portable applications such as ShareGate. The product is great! We have tested it, but not with a lot of data. Please check this tool out for your next migration and tell them we sent you!

Metalogix has been a key player in SharePoint since we can remember. We wrote up some of this information a while back with the first release of the product, and you definitely need to try this product. Although we had a bad experience with it, we have heard good things about the tool now and the features are very impressive. The downside to it is that is not very user-friendly, so there is a learning curve. Also this tool, despite its improvement, does not support PowerShell integration. We just wanted to update you on this tool to make it clear that you should try it.

Microsoft also has a free migration tool that allows you to migrate to Microsoft 365 from SharePoint On-Premises platforms, including 2016, 2013 including Foundation, and 2010 including Foundation. It also includes hooks for connecting to the network and local file shares. To find out more about this tool, please go to https://docs.microsoft.com/en-us/sharepointmigration/introducing-the-sharepoint-migration-tool.

If you are looking for a planning and assessment tool, Microsoft provides the SharePoint Migration Assessment Tool (SMAT), a command-line tool that scans SharePoint farms to identify issues before you migrate. This will give you a report so that you can go back and fix any issues before migration. This tool can also be found in the preceding link.

Managed metadata: This is a more complex migration. Some of the migration tools that are available allow you to migrate metadata using the methods provided by the tool. The issue with managed metadata is that there is a GUID for each term set, and the term within the service will not change if you migrate the data. So, if you are doing a service and content database migration, this method of migrating the databases works because there is no GUID change. If you cannot get these methods working and you must do a new installation, you will have to recreate the service and the terms within it.

When performing a content database migration, these details for the managed metadata can be brought over by backing up and restoring the managed metadata service database. When you’re creating your managed metadata service, you will use that restored backup to create the new service on the new farm. If you are using a content type hub, update that link using the following PowerShell script:

Set-SPMetadataServiceApplication -Identity "Managed metadata
Service Application" -HubURI "http://sitename/contenttypehub"

If this is not the method you would like to use to restore the service database, you can use the export-import method. When you migrate the service in this way, you will have to create the service first and then import the information using Excel. You can do that using the following PowerShell script:

Add-PSSnapin Microsoft.SharePoint.Powershell $metadataApp= Get-
SpServiceApplication | ? {$_.TypeName -eq "Managed metadata
Service"} $mmsAppId = $metadataApp.Id $mmsproxy = Get-SPServiceApplicationProxy | ?{$_.TypeName -eq "Managed metadata
Service Connection"} Export-SPMetadataWebServicePartitionData
-Identity $mmsAppId -ServiceProxy $mmsproxy -Path "C:\MMD.cab"

Important note

Make sure you do not have any duplicates in your file before moving forward. If you have duplicates, the process will error on import.

The script for importing is as follows:

Add-PSSnapin Microsoft.SharePoint.Powershell $metadataApp=
Get-SpServiceApplication | ? {$_.TypeName -eq "Managed
metadata Service"} $mmsAppId = $metadataApp.Id $mmsproxy =
Get-SPServiceApplicationProxy | ?{$_.TypeName -eq "Managed
metadata Service Connection"} Import-SPMetadataWebServicePartitionData -Identity $mmsAppId -ServiceProxy $mmsproxy -Path "//
SharedPath/MMD.cab"

Remember that if you were using managed metadata previously in your farm, then you need to migrate this service database and create a service application first before migrating any content. The content, when migrated, will find out whether this data is available so that it can populate the data in the fields that use metadata columns. In some cases, it may give you the functionality to migrate this metadata, depending on the tool you are using. When performing content database migrations, you will have to move the metadata first.

You also need to check that there is a Content Type Hub in your past farm. If so, you will want to make sure you configure that in the new SharePoint 2019 farm. You can do this by setting up a new site collection solely for this purpose and adding the URL to the Content Type Hub form field when editing your Managed Metadata Service application. You can also update the Content Type Hub using PowerShell, as shown in the preceding script, to create the service application. If you have already created it without it, use this script:

Set-SPMetadataServiceApplication -Identity  -HubURI

As you can see, there is a sequence you must follow when performing migrations, and scheduling is a big part of this process. Next, we will learn how and when to schedule migrations.

Scheduling migrations

At this point, everyone knows they are moving to the new SharePoint environment. Everyone is on edge to do this because, at the end of the day, most are afraid of change. So, how do we schedule migrations? Our take on this is that you start with the departments that are most eager to jump in. The reason you want to do this is that you will get the most help with the process from resources within the department. This creates a team effort that you might not get from another department, which could defeat the migration before it starts.

If you don’t have any eager departments, which we doubt you will, take one department that you have some camaraderie with or one of the hardest you will have to move due to complexity, and test their migration first. You can at least test migration with this first department and have the users test the migrated content and compare it between the source and the destination. You should be able to give them an error report from any migration type and work with them on how you will mitigate the errors when performing the final migration of the content.

In the Types of migration section, we discussed that migration tools have some scheduling automation capabilities that you can use to configure a migration and create a scheduled task when a particular content migration should run. You can also migrate using PowerShell by scheduling a command to execute on the server at a certain time, but most tools have this integrated with the product.

When running scheduled migrations, you do not have to be present. You could migrate several departments at a time if the content size is reasonable. This also depends on what your time frame is because you want to make sure your users have access to the content you’re migrating, maybe the next morning or after the weekend on Monday morning.

You will need to support the migration from time to time to monitor the progress of the migrations you are scheduling. Believe us, there are always errors. To help with such errors, you may need to assign tasks to people in your department to help monitor their progress. If you have tested your migrations and are pretty sure you have done the testing well, or if you know there is not much complexity in the site you’re migrating, then you may be able to let the scheduled migration run on its own.

If you were testing and used the most difficult customer based on the complexity of the content, then this would have given you a realistic look at the tool and any errors that come up, based on this content. Always take this on with all resources on the ground. If it is just you, get support from Microsoft, a third-party company, or from the company that owns the tools you purchased. This can be done successfully if you plan out the migration properly and identify any errors and mitigate them prior to a test migration.

Again, this may take a couple of test migrations, but the mitigation of errors that surface is the key to success and how quickly you can turn them around. Make sure you always take account of the time you may have to spend on the backend recreating content in a site. Again, if the solution is not available in the next version of the farm, you have to make sure you recreate it using a new tool and communicate the difficulty and/or discontinuation of the product to the customer.

Important note

If you have a couple or even a few web applications, NEVER EVER migrate one web application if the services on the new farm are not stable. If you are still in configuration mode and making updates to services, DO NOT migrate anyone to that unstable farm. Once you have a stable service platform and you have tested those services, you can migrate your first set of user content.

Concluding remarks

Success is the best way to build confidence in the company as you undertake migrations. Choose the right department to work with and test your migration. Review your errors and figure out how to fix them before the final migrations. This may take another test migration, but do not let the first migration fail. Do your best to be successful and have the department using the new environment brag about how easy it was to migrate so that others will be at ease and want to move immediately.

To find out how to bring on-premise and cloud collaboration features to life with Microsoft’s SharePoint Server, check out the book Implementing Microsoft SharePoint 2019 by Lewin Wanzer and Angel Wood.

What is Moodle Workplace?

Packt — Fri, 23 Apr 2021 10:02:23 GMT

Moodle is the most-used Learning Management Systems (LMS) there is, and now it’s joined by Moodle Workplace. In this overview, we introduce Moodle Workplace and describe its various new key features. We then compare this newly updated software with standard Moodle, its little sibling, and with Totara, the other popular Moodle distribution for learning in organizations and businesses. Finally, we explore Moodle Workplace’s business model and its versioning policy.

Introducing Moodle Workplace

An open-source LMS that has been developed by Moodle Pty Ltd, Moodle Workplace is based on standard Moodle and has extra features specially designed for corporate and organizational training, resulting in a powerful and flexible platform for workplace learning (https://moodle.com/workplace).

The main targets for Workplace are companies — from small to medium enterprises, all the way to global corporations and governments (regional and national), as well as public-sector and non-governmental organizations such as charities, foundations, and healthcare providers. However, Moodle Workplace is not limited to these organizations: a university might make use of Moodle Workplace for the training and development of its employees while using standard Moodle as their LMS for students.

The key features of Moodle Workplace are as follows:

Multi-tenancy

Allows you to create different tenants in the same Workplace installation. These are completely isolated from each other with their own look and feel, structure, users, and learning entities.

There might be multiple hospitals belonging to the same trust, but each with their individual corporate identity and their own employees organized in their local department structure. There is likely to be learning content unique to the hospital’s specialism, but also courses that are shared across hospitals.

Organization Structure

Allows you to model your organization with multiple hierarchical department and position frameworks, and define reporting lines assigning jobs to your staff, giving managers visibility on learning progress.

The departments of a heart clinic might be Cardiac Imaging and Testing, General Cardiology, Heart Rhythm/Electrophysiology, Interventional Cardiology, and Peripheral Vascular Disease Treatment. Typical positions are Cardiologist, Electrophysiologist, and Cardiac Nurse. A job assignment would assign user Joe Flutter to the position of Cardiac Nurse in the Department of Electrophysiology, which is managed by user Joanne Ticker.

Report Builder

Allows you to build and customize your own reports with drag’n’drop, instant preview, inline column editing, groupings, aggregation, and restriction to specific audiences from various data sources, including the internal Workplace datastore.

A typical report would display all staff who have completed the course Clinical Documentation and can be filtered by department and position.

Dynamic Rules

Allows you to define and execute centralized and automated rules using an ‘if this then that’ conditional approach to trigger actions when certain conditions are met.

A sample rule might stipulate that upon course completion of Clinical Documentation, a message is sent to his/her manager and a certificate of completion is issued alongside a badge.

Programs

Allows you to establish learning pathways for your employees by adding a combination of courses or a hierarchical sequence of courses. You have the tools to control what learners need to complete to progress and ensure depth understanding of subjects by defining different completion criteria at the program and set levels.

A typical program comprises a number of courses that make up an onboarding program for a particular audience, for instance, nursing apprentices. The assignment to a program can take place via dynamic rules, and reporting shows the progress of all recent hires.

Certifications

Allows you to validate learning paths by offering certifications for recurring programs. Ensure your staff is compliant, enabling certification expiry and re-certification pathways.

Certifications are used for compliance training. Typical compulsory training with a validity period in the hospital context are health & safety and hygiene sessions.

Certificates

Allows you to issue certificates to learners or teams in any context: side-wide, per tenant, by category or course. The creation of diplomas takes place in a built-in certificate designer supporting digital signatures.

Certificates can either be used as incentives for hospital employees, or they become part of the staff’s employment record, in particular when there is a legal requirement to complete a course, program, or certification.

Appointment Bookings

Allows you to create one-to-one meetings or seminars with multiple sessions through its internal booking system. Session management supports bulk events creation, attendance handling, and wait-listing features.

A ward manager might schedule regular 1:1 sessions with all nurses to discuss any issues in the facility. Both scheduling and attendance recording takes place via the Appointment bookings module.

Mobile App

Allows you to provide your employees with a seamless e-learning experience across devices and allow your learners to fit learning around their life. A fully branded Workplace App supports your corporate identity for a fully customized experience.

Now that you are familiar with the key features of Moodle Workplace, let’s compare it with its little brother, standard Moodle.

Comparing Moodle Workplace and standard Moodle

Use of the term Moodle — both as a noun and a verb — has been synonymous with the leading open source LMS. However, there are multiple components in the Moodle product family, and the one almost everyone refers to as Moodle is technically called Moodle Core. The high-level components can be seen in the following diagram:

Figure 1 - Moodle Core versus Moodle Workplace

Let’s have a closer look at each of the components in the preceding diagram.

Moodle Core or standard Moodle

The original open-source LMS, Moodle has become the de facto standard worldwide in educational settings, particularly in schools, colleges, and universities. Since its inaugural launch in 2002, Moodle Core has become the benchmark that every LMS is measured against. It has won a wide range of international accolades and established itself as an ecosystem for numerous educational tools and services.

Important note

We are going to refer to Moodle Core as standard Moodle for simplicity and better readability.

Standard Moodle contains facilities for formative and informal assessment, synchronous and asynchronous communication and collaboration, grading, competencies, and much more. Various standards, such as SCORM, LTI, and IMS, have been adopted, and integration via numerous standard protocols is available, including SOAP and REST web services, Active Directory, and SAML.

A key ingredient to the success story of Moodle is its flexibility and customizability. In addition to hundreds of application configuration options, various elements can be tailored to your needs, including the look and feel of the LMS to represent the corporate identity of your organization. Think of Moodle as a massive box of Lego bricks, where you can either follow pre-built sets or build your own individual system. A major contribution to the latter is Moodle plugins.

Moodle plugins and Moodle LMS

While standard Moodle is a powerful and comprehensive LMS, very few sites solely rely on the base system. Instead, add-on modules are used to supplement the feature set of standard Moodle to customize the platform to individual requirements. These add-ons are called Moodle plugins and are mostly community-contributed additions to standard Moodle, extending its functionality for a specific use case. At the time of writing, there are over 1,750 (!) entries in the official Moodle plugins database at https://moodle.org/plugins.

There exist a plethora of third-party Moodle plugins that add new functionalities, fix problems, or integrate Moodle with external systems and cloud services. This also covers add-ons for commercial software popular in workplace settings, such as the following, for example:

Office systems: Microsoft has developed a suite of plugins to allow Microsoft Office 365 usage within Moodle. This includes logging in via OpenID, access to OneDrive, integration with Office resources, and Outlook calendar synchronization.
Web conferencing: Various commercial web conferencing suppliers provide plugins for their system to be used from within a Moodle course. Examples are WebEx, Zoom, and BigBlueButton.
Video platforms: Streaming is a highly effective way to transport video content, which is already supported via basic integrations with YouTube and Vimeo. The Moodle functionality for the latter can be nicely enhanced using the popular Video Time plugin. Additionally, dedicated video platforms such as Kaltura also provide suites of plugins for smooth integration with Moodle.

Moodle LMS is effectively standard Moodle plus — optionally — one or many Moodle plugins.

Workplace plugins

Moodle has developed a set of Workplace plugins that contain the key features of Moodle Workplace that sit on top of Moodle LMS. All Moodle plugins still work in Moodle Workplace, with the rare exception of features that conflict with Workplace functionality.

Some of the Moodle Workplace plugins will make their way into standard Moodle over time, so Moodle LMS users can benefit from the investment that has gone into the Enterprise Edition of Moodle. The priority and timing will be determined through a collaborative approach with the Moodle community and Moodle Partners.

This section has given you a brief overview of the delta between standard Moodle and Moodle Workplace. Since Workplace is an extension of Moodle LMS, the differences are clear cut. This is different when it comes to Totara Learn, which we will cover in the next section.

Comparing Moodle Workplace and Totara Learn

Totara LMS is a subscription-based Moodle distribution that was launched in 2010 by Totara Learning (https://totaralearning.com). Initially, it was an extension to standard Moodle, plus some plugins, that was kept in lockstep with Moodle’s releases until version 2.9. Moodle introduced competencies and learning plans in version 3.1, which conflicted with their counterparts in Totara. Totara LMS was renamed Totara Learn; the latest version is Totara Learn 13.

Totara Learn is now part of a product family — Totara Experience Platform (TXP) — that comprises the following three interlinked components:

Totara Learn: An LMS for training and development

Totara Learn is Totara’s flagship product and has been the only serious contender among the open-source Moodle LMS distributions for the commercial sector until Moodle Workplace arrived on the scene. Totara Learn is the product in the Totara suite that is used for comparison with Moodle Workplace in this article.

Key features include online learning design and delivery, offline seminar management, assessment and certification, learning plans, program management, adaptive learning, and reporting.

Totara Engage: A Learning Experience System (LXP) focused on social learning

Totara Engage provides a secure social space powered by people, where learning comes recommended and self-directed, and not just mandated by the organization. Totara Engage is an LXP that supports day-to-day workplace communication, knowledge sharing, and discovery.

Key features include peer-to-peer content creation and sharing, collaborative workspaces, integration with Microsoft Teams and Slack, informal learning, messaging, discussion, recognition, pulse surveys, and action lists.

Totara Perform: A performance management tool supporting various HR activities

Totara Perform is a flexible performance platform that allows organizations to proactively manage staff performance in order to operate more effectively and achieve their goals.

Key features include appraisals, check-ins, goals, OKRs, 360° feedback, competencies, evidence banks, and reporting dashboards.

Additional products might be added to the mix in the future, but for now, these are the available components. All three products make use of the same underlying core business logic represented in a shared services layer.

Totara Learn and Moodle Workplace target the same audiences and are effectively competing for products. You may ask what the differences are between the two available offerings. Generally, the dissimilarities can be grouped into three categories:

Features unique to Moodle Workplace
Features unique to Totara Learn
Features that exist in both products but are implemented differently

Let’s take a look at these in more detail.

Features unique to Moodle Workplace

There are several features that only exist in Moodle Workplace that are not offered by Totara Learn. An example is dynamic rules, where automation can be achieved via ‘if this, then that’ rules to trigger actions when certain conditions are met. For instance, when a course has been completed, a certificate will be issued, and a notification will be sent to the user and also to the responsible manager.

Totara Learn comes with so-called dynamic audiences, which support similar workflows, but are nowhere near as comprehensive or flexible as Workplace’s dynamic rules.

Another example is the migration tool, which supports the export and import of various Workplace elements into other Workplace instances.

Features unique to Totara Learn

Equally, some features are unique to Totara Learn. For instance, seminar management is a component covering room and equipment management, manager approval, sign-ups to the course catalog, and much more.

Moodle Workplace contains the appointments feature mentioned earlier, which covers some of the same use cases, but to date, it is no match for Totara’s seminar management functionality.

Another example is the highly customizable course catalog offered by Totara Learn. It allows users to browse, search, and filter courses, programs, and certifications. Moodle Workplace currently doesn’t have a catalog feature.

Features that exist differently in both products

Competencies and learning plans have already been mentioned as two features that exist in both products but have been modeled and implemented very differently.

Another good example is the way managers are modeled. In Totara Learn, a manager has a 1:1 relationship with a user; that is, one user is the manager of another user. Both users belong to one or many organizations. In Moodle Workplace, a user belongs to a department, and a manager is responsible for this organizational entity. Every user that belongs to the same department reports to the manager in charge. The result — one user is superior to one or many other users — is the same in both systems, but their implications when implementing user-profiles and synchronizing HR data are significant.

There are plenty of features that belong to this category but a full comparison is beyond the scope of this overview. Instead, we offer some pointers on how to decide which is the better product for your setup.

So, which is the better product?

The only way to answer this question is that it depends on the project at hand and its requirements. For instance, if your organization only provides sporadic face-to-face training, Moodle Workplace is likely to suffice. However, if you require full-blown seminar management, Totara Learn might be the better option. A thorough evaluation of each element and how it responds to your requirements is highly recommended to make an informed decision.

It must be noted that both systems continue to grow organically at a breakneck pace. Also, both product architects look sideways to see what the other side is doing, and we all know that competition is good for business!

Both Moodle and Totara offer their services through global partner networks. While the principles behind the partner models are similar, the business models are very different.

The Moodle Workplace business model

60% of Moodle’s revenue comes from the sectors targeted by Moodle Workplace, that is, commercial, not-for-profit, and government. Well-known examples of organizations that run Moodle include global enterprises such as Google, Cisco, Coca-Cola, Shell, Intel, and Bridgestone, but also large organizations such as the United Nations, which relies on Moodle as its LMS.

Most of Moodle’s revenue comes from Moodle Partners (Standard, Premium, and Integration), who pay royalties on all Moodle work. Customers who (hopefully) engage with a Moodle Partner pay fees for the services provided, such as consultancy, hosting, support, custom development, theme creation, and integration work. These high-level revenue streams are displayed in the following diagram, showing the overall Moodle business model:

Figure 2 - Moodle business model

Moodle HQ employs a global team of staff who develop and maintain a range of offerings, comprising the following six products (see https://moodle.com/products for more details):

Moodle LMS: Standard Moodle plus Moodle plugins (as covered previously).
Moodle App: This includes the Moodle app, the Moodle Workplace app, and related chargeable branding services.
MoodleCloud: A Moodle Software-as-a-Service (SaaS) offering of Moodle LMS and Moodle Workplace. There is a free basic plan for small Moodle LMS projects; all other packages are paid-for offerings.
Moodle Workplace: Well, you know what this is…
MoodleNet: An open-source social media platform for educators, integrated with Moodle and focused on collaboratively curating collections of open content.
Moodle Education: The Moodle Educator Certification (MEC) program is a comprehensive teaching and learning curriculum designed to help you develop transferable knowledge and skills to be effective educators in today’s growing digital workplace.

Both standard Moodle and Moodle Workplace are open source under the GNU Public License 3.0. However, unlike standard Moodle, which is available from the download section of https://moodle.org, Moodle Workplace is currently only available through Premium Moodle Partners (https://moodle.com/partners). Workplace can either be hosted in the data center facilities of a Moodle Partner or on-premises. The latter requires you to sign a three-way agreement between Moodle HQ, your Moodle (Premium) Partner, and your company. You need to agree to the Moodle Workplace License, which outlines your responsibilities as the customer.

The purpose of the aforementioned arrangement is to protect Moodle’s significant investment in Moodle Workplace and to ensure that the Workplace source code is not distributed to any third parties.

The other option for utilizing Moodle Workplace in your business is via Workplace on MoodleCloud, a SaaS offering from Moodle HQ. It is a turnkey solution for sites of up to 1,000 users that includes a video conferencing tool (BigBlueButton) as well as access to content from GO1. However, Workplace on MoodleCloud does not allow you to install plugins or themes, and certain features and configuration settings cannot be changed. Depending on the chosen package, different Workplace features are limited, for instance, the number of tenants allowed or the number of custom reports that can be created. For more details and pricing, contact your Moodle Partner.

Now that you are familiar with Moodle’s business model and its primary revenue streams, let’s have a look at the Moodle Workplace versions, which differ slightly from the release schedule of standard Moodle.

Understanding Moodle Workplace versions

Moodle Workplace versions are based on the releases of standard Moodle, so Moodle Workplace 3.9.2 is based on standard Moodle 3.9.2. Minor versions of Workplace are released the day after their standard Moodle counterparts, which typically takes place on the second Monday of odd months (January, March, May, July, September, and November). The same applies to unscheduled releases in the case of a severe security issue or serious regression being fixed. Major versions of Workplace are released 1–2 weeks after the respective major version of standard Moodle. The release frequency is twice a year (the second Mondays of May and November).

Important note

Moodle Workplace is always based on the official update (minor version) that does not have a ‘+’ sign.

Currently, backporting to older releases is not supported by Workplace, even if the standard Moodle counterpart is still under support. However, this is likely to change in the future. The following timeline diagram demonstrates the interdependence between standard Moodle and Moodle Workplace releases:

Figure 3 - Release timeline for standard Moodle versus Moodle Workplace

Version 3.7 was the first (mostly internal) version of Moodle Workplace. Version 3.9.2 is the most up-to-date release at the time of writing. Standard Moodle 3.5 and 3.9 are Long-Term Support (LTS) releases that will be supported for 3 and 5 years respectively. The next major release of standard Moodle will be version 4.0, to be released in November 2021.

For more information on standard Moodle releases, check out https://docs.moodle.org/dev/Releases. You can find the release notes of Moodle Workplace at https://docs.moodle.org/en/Moodle_Workplace_Release_notes.

Each Moodle Workplace release is a full distribution that contains standard Moodle, all Workplace plugins including the Workplace theme, and several core modifications that can be either future core fixes or Workplace-only patches that allow Workplace plugins to hook into the Core functionality. That is, when installing Moodle Workplace, you do not have to install standard Moodle first and then add plugins or apply Core patches.

This section has provided you with an overview of the Moodle Workplace release schedule and its relationship with the standard Moodle releases. This timetable should be taken into account when planning updates and upgrades, as this is likely to involve maintenance work as well as some downtime.

Summary

In this article, you have learned about the key features of Moodle Workplace, such as multi-tenancy, dynamic rules, reports, and many more. You have also learned about the divergence between standard Moodle and Moodle Workplace, which is a powerful extension of the base product. The design and development of this delta between the two products is quite a technical masterpiece, maintaining 100% compatibility with standard Moodle while adding a plethora of features that can be used consistently throughout the system.

We looked at the popular Totara product suite and saw that important differences exist between Moodle Workplace and Totara Learn, both in terms of functionality and commercials. This led to a discussion of Workplace’s business model, showing its main revenue streams and stressing the importance of Moodle Partners. Finally, you gained some insight into the Workplace versioning policy and its dependencies on standard Moodle.

The depth and breadth of Moodle Workplace give it real clout as a learning solution for training and development. Yes, Moodle was a bit late to the party, but when it finally turned up, it brought a superstar of a guest along.

To learn more about what Moodle has to offer in the fast-growing online learning space and how to adopt Moodle Workplace in your organization, check out Alex Büchner’s book Corporate Learning with Moodle Workplace.

What Is Cyber Threat Intelligence?

Packt — Tue, 20 Apr 2021 08:43:27 GMT

The objective of this overview is to help you become familiar with some of the main concepts and terminology used in the field of cyber threat intelligence (CTI). We shall cover the following topics:

Cyber threat intelligence
The intelligence cycle
Defining your intelligence requirements
The collection process
Processing and exploitation
Bias and analysis

With no further ado, let’s get started!

Cyber threat intelligence

If we want to discuss the roots of intelligence discipline, we could probably go back as far as the 19th century, when the first military intelligence departments were founded. We could even argue that the practice of intelligence is as old as warfare itself and that the history of humanity is full of espionage stories as a result of needing to have the upper hand over the enemy.

It has been stated over and over that in order to have a military advantage, we must be capable not only of understanding ourselves but also the enemy: how do they think? How many resources do they have? What forces do they have? What is their ultimate goal?

This military need, especially during the two World Wars, led to the growth and evolution of the intelligence field as we know it. Several books and papers have been written about the craft of intelligence, and I sincerely encourage anyone interested in the matter to explore the resources available on the CIA website (https://www.cia.gov/).

The definition of intelligence has been under academic discussion among people better-versed in the matter than me for more than two decades. Unfortunately, there is no consensus over the definition of intelligence practice. We shall use the definition proposed by Allan Breakspear in his paper A New Definition of Intelligence (2012) as a reference:

Intelligence is a corporate capability to forecast change in time to do something about it. The capability involves foresight and insight, and is intended to identify impending change, which may be positive, representing opportunity, or negative, representing threat.

Based on this, we are going to define CTI as a cybersecurity discipline that attempts to be a proactive measure of computer and network security, which nourishes itself from the traditional intelligence theory.

CTI focuses on data collection and information analysis so that we can gain a better understanding of the security threats facing an organization. This helps us protect its assets. The objective of any CTI analyst is to produce and deliver relevant, accurate, and timely curated information — that is, intelligence — so that the recipient organization can learn how to protect itself from a potential threat.

The sum of related data generates information that, through analysis, is transformed into intelligence. However, as we stated previously, intelligence only has value if it is relevant, accurate, and, most importantly, if it is delivered on time. The purpose of intelligence is to serve those responsible for making decisions so they can do so in an informed way. There is no use for this if it is not delivered before the decision must be made.

This means that when we talk about intelligence, we are not only referring to the product itself, but also to all the processes that make the product possible. We will cover this in considerable detail.

Finally, we can classify intelligence according to the time that’s been dedicated to studying a specific subject, either by distinguishing between long-term and short-term intelligence, or according to its form; that is, strategic, tactical, or operational intelligence. In this case, the intelligence that’s delivered will vary, depending on which recipients are going to receive it.

Strategic level

Strategic intelligence informs the top decision-makers — usually called the C-suite: CEO, CFO, COO, CIO, CSO, CISO — and any other chief executive to whom the information could be relevant. The intelligence that’s delivered at this level must help the decision-makers understand the threat they are up against. The decision-makers should get a proper sense of what the main threat capabilities and motivations are (disruption, theft of proprietary information, financial gain, and so on), their probability of being a target, and the possible consequences of this.

Operational level

Operational intelligence is given to those making day-to-day decisions; that is, those who are in charge of defining priorities and allocating resources. To complete these tasks more efficiently, the intelligence team should provide them with information regarding which groups may target the organization and which ones have been the most recently active.

The deliverable might include CVEs (common vulnerabilities and exposures) and information regarding the tactic used by, as well as the techniques of, the possible threat. For example, this could be used to assess the urgency to patch certain systems or to add new security layers that will hinder access to them, among other things.

Tactical level

Tactical intelligence should be delivered to those in need of instantaneous information. The recipients should have a complete understanding of what adversary behaviors they should be paying attention to in order to identify the threats that could target the organization.

In this case, the deliverable may include IP addresses, domains and URLs, hashes, registry keys, email artifacts, and more. For example, these could be used to provide context to an alert and evaluate if it is worth involving the incident response (IR) team.

So far, we have defined the concepts surrounding intelligence, CTI, and intelligence levels, but what do we understand by the term threat in the cyber realm?

We define a threat as any circumstance or event that has the potential to exploit vulnerabilities and negatively impact operations, assets (including information and information systems), individuals, and other organizations or societies of an entity.

We could say that the main areas of interest for cyber threat intelligence are cybercrime, cyberterrorism, hacktivism, and cyberespionage. All of these can be roughly defined as organized groups that use technology to infiltrate public and private organizations and governments to steal proprietary information or cause damage to their assets. However, this doesn’t mean that other types of threats, such as criminals or insiders, are outside the scope of interest.

Sometimes, the terms threat actor and advanced persistent threat (APT) are used interchangeably, but the truth is that although we can say that every APT is a threat actor, not every threat actor is advanced or persistent. What distinguishes an APT from a threat actor is its high level of operational security (OPSEC), combined with a low detection rate and a high level of success. Keep in mind that this might not apply perfectly to all APT groups. For example, there are some groups that feed on the propaganda from the attack, so they put a lower effort into not being identified.

In order to generate valuable intelligence, it is important to work with clear and defined concepts so that you can structure the data and generate information. It is not mandatory to choose an existing terminology, but the MITRE Corporation has developed the Structured Threat Information Expression (STIX) (https://oasis-open.github.io/cti-documentation/) in order to facilitate the standardization and sharing of threat intelligence.

So, if we follow the STIX definition (https://stixproject.github.io/data-model/), threat actors are “actual individuals, groups, or organizations believed to be operating with malicious intent.” Any threat actor can be defined by any of the following:

Its type (https://stixproject.github.io/data-model/1.1/stixVocabs/ThreatActorTypeVocab-1.0/)
Its motivations (https://stixproject.github.io/data-model/1.1/stixVocabs/MotivationVocab-1.1/)
Its sophistication level (https://stixproject.github.io/data-model/1.1/stixVocabs/ThreatActorSophisticationVocab-1.0/)
Its intended effect (https://stixproject.github.io/data-model/1.1/stixVocabs/IntendedEffectVocab-1.0/)
The campaigns it was involved in
Its Tactics, Techniques, and Procedures (TTPs) (https://stixproject.github.io/data-model/1.2/ttp/TTPType/)

In summary, cyber threat intelligence is a tool that should be used to gain better insight into a threat actor’s interests and capabilities. It should be used to inform all the teams involved in securing and directing the organization.

To generate good cyber intelligence, it is necessary to define the right set of requirements for understanding the needs of the organization. Once this first step has been accomplished, we can prioritize the threats the team should be focusing on and start monitoring those threat actors that might have the organization among its desired targets. Avoiding the collection of unnecessary data will help us allocate more time and resources, as well as set our primary focus on the threats that are more imminent to the organization.

As Katie Nickels stated in her talk The Cycle of Cyber Threat Intelligence (2019, https://www.youtube.com/watch?v=J7e74QLVxCk), the CTI team is going to be influenced by where they’ve been placed, so having them at a central position in the structure of the organization will help the team actually support different functions. This can be visualized as follows:

Figure 1 - CTI team center role

We will now look at the intelligence cycle.

The intelligence cycle

Before we dive into the theory of the intelligence cycle, I believe it is worth showing the relationship between data, knowledge, and intelligence practice through what is known as a knowledge pyramid. In it, we can see how the facts, through measurement, are transformed into data that we can extract information from when processing it. When analyzed together, it can be transformed into knowledge. This knowledge interacts with our own experience and forms the basis of what we call wisdom. It is this ultimate wisdom that we rely on for decision-making.

As shown in the following pyramid, we can intertwine this knowledge pyramid with the processes that are part of what is widely known as the intelligence cycle:

Figure 2 - DIKW pyramid

In short, we can deduce that an intelligence analyst must process data to transform it into wisdom (intelligence), which in the last instance will lead to an action (decision).

Traditionally, the intelligence process is understood as a six-phase cycle: planning and targeting, preparation and collection, processing and exploitation, analysis and production, dissemination and integration, and evaluation and feedback. Each of these phases presents its own particularities and challenges:

Figure 3 - The intelligence cycle

We will now look at each of these phases in detail.

Planning and targeting

The first step is to identify the intelligence requirements (IRs). Any information that the decision-makers need and don’t know enough about falls under this category.

In this stage of the process, it is important to identify the key assets of the organization, why the organization might be an interesting target, and what the security concerns of those in charge of making decisions are.

It’s also important to identify the potential threats that exist and what mitigations can be prioritized (through a process known as threat modeling), as well as establishing a collection framework and collection priorities.

Preparation and collection

This stage refers to defining and developing collection methods to obtain information regarding the requirements that were established in the previous phase.

It is important to keep in mind that it’s impossible to answer all the questions we may have and meet all our intelligence requirements.

Processing and exploitation

Once the planned data has been collected, the next step is to process it to generate information. The processing method is usually not perfect, and the amount of data that the intelligence team is able to process is always lower than the amount of data that has been gathered. All data that does not get processed is the same as data not collected at all. It’s lost intelligence.

Analysis and production

The information that’s been gathered so far must be analyzed in order to generate intelligence. There are several techniques that are used for intelligence analysis and to prevent the analyst’s bias. The cyber threat intelligence analyst must learn how to filter their personal views and opinions to carry out the analysis.

Dissemination and integration

In this stage, the intelligence that’s been produced is distributed to the necessary sectors. Before distribution, the analysts have to consider a variety of things, such as what the most pressing issues are among the intelligence that’s been collected, who should receive the report, how urgent the intelligence is or how much detail the recipient needs if the report should include preventive recommendations, and so on. Sometimes, different reports may need to be created and directed to different audiences.

Evaluation and feedback

This is the final stage of the process and probably the most difficult to achieve, mainly due to the usual lack of feedback from intelligence recipients. Establishing good mechanisms to get feedback helps intelligence producers evaluate the effectiveness of the intelligence that’s been generated before they repeat the process over and over, without making the necessary adjustments that will make the intelligence that’s produced more relevant to the recipients. As intelligence producers, we want our intelligence to be relevant — we want our intelligence to help the decision-makers to make informed decisions. Without gathering the appropriate feedback, we won’t know if we are achieving our goal, and we won’t know which steps to take to improve our product.

This model has been widely accepted and adopted, especially in the United States of America and among those who follow their academic discussions in an attempt to replicate its methods. Despite this wide acceptance, there have been some vocal criticisms against this model.

Some have pointed out that the current model depends excessively on the data that’s been collected, and also that technological advances have allowed us to collect massive amounts of it. This endless harvesting process and the capacity to better represent the data that’s been collected leads us to believe that this process is enough for us to understand what is happening.

There have been alternative proposals for the intelligence cycle. For anyone interested in studying more on this, a particularly interesting contribution has been published by Davies, Gustafson, and Ridgen (2013) titled The Intelligence Cycle is Dead, Long Live the Intelligence Cycle: Rethinking Intelligence Fundamentals for a New Intelligence Doctrine (https://bura.brunel.ac.uk/bitstream/2438/11901/3/Fulltext.pdf), in which what has been labeled the UK Intelligence Cycle is described in detail:

Figure 4 - The Core Functions of Intelligence (JDP 2–00) (3rd Edition)

Now, let’s learn how to define and identify our intelligence requirements.

Defining your intelligence requirements

As defined by the United States Department of Defense, an intelligence requirement (IR) is as follows:

1. Any subject, general or specific, upon which there is a need for the collection of information, or the production of intelligence.

2. A requirement for intelligence to fill a gap in the command´s knowledge or understanding of the battlespace or threat forces.

The first stage in the intelligence cycle is to identify the information that the decision-maker needs. These requirements should be the driving factor in the intelligence team’s collection, processing, and analysis phases.

The main problem that occurs when identifying these IRs is that, usually, the decision-makers do not know what information they want until they need it. Moreover, other issues, such as resource and budget shortcuts or sociopolitical events, may arise, as well as difficult the task of identifying and satisfying the IRs.

Posing and trying to answer a series of questions, not only the ones stated here as examples, could be a good starting point when you’re trying to identify the PIRs (P for priority, referring to those that are more critical) and the IRs of an organization.

Important note

Identifying Intelligence Requirements

When working out your intelligence requirements, ask yourself the following questions:

What’s the mission of my organization?

What threat actors are interested in my organization’s industry?

What threat actors are known for targeting my area of operation?

What threat actors could target my organization in order to reach another company I supply a service for?

Had my organization been targeted previously? If so, what type of threat actor did it? What were its motivations?

What asset does my organization need to protect?

What type of exploits should my organization be looking out for?

There are four criteria to keep in mind when validating a PIR: the specificity and the necessity of the question, the feasibility of the collection, and the timeliness of the intelligence that would be generated from it. If the requirement meets all these criteria, we can start the collection process around it. In the next section, we will cover this in detail.

The collection process

Once the intelligence requirements have been defined, we can proceed with collecting the raw data we need to fulfill them. For this process, we can consult two types of sources: internal sources (such as networks and endpoints) and external sources (such as blogs, threat intelligence feeds, threat reports, public databases, forums, and so on).

The most effective way to carry on the collection process is to use a collection management framework (CMF). Using CMF allows you to identify data sources and easily track the type of information you are gathering for each. It can also be of use to rate the data that’s been obtained from the source, including how long that data has been stored, and to track how trustworthy and complete the source is. It is advised that you use the CMF to track not only the external sources but also the internal ones. Here’s an example of what one would look like:

Figure 5 - Simple CMF example

Dragos analysts Lee, Miller, and Stacey wrote an interesting paper (https://dragos.com/wp-content/uploads/CMF_For_ICS.pdf?hsCtaTracking=1b2b0c29-2196-4ebd-a68c-5099dea41ff6|27c19e1c-0374-490d-92f9-b9dcf071f9b5) about using CMF to explore different methodologies and examples.

Another great resource available that can be used to design an advanced collection process is the Collection Management Implementation Framework (https://studylib.net/doc/13115770/collection-management-implementationframework-what-does-...), designed by the Software Engineering Institute.

Indicators of compromise

So far, we’ve talked about finding the intelligence requirements and how to use the CMF. But what data are we are going to collect?

An indicator of compromise (IOC), as the name suggests, is an artifact that’s been observed in a network or in an operating system that, with high confidence, indicates that it has been compromised. This forensic data is used to understand what happened, but if collected properly, it can also be used to prevent or detect ongoing breaches.

Typical IOCs may include hashes of malicious files, URLs, domains, IPS, paths, filenames, Registry Keys, and malware files themselves.

It is important to remember that, in order to be really useful, it is necessary to provide context for the IOCs that have been collected. Here, we can follow the mantra quality over quantity — a huge number of IOCs does not always mean better data.

Understanding malware

Malware, short for malicious software, is not everything, but it can be an incredibly valuable source of information. Before we look at the different types of malware, it is important for us to understand how the malware typically works. Here, we need to introduce two concepts: the dropper and the Command and Control (C2 or C2C).

A dropper is a special type of software designed to install a piece of malware. We will sometimes talk about single-stage and two-stage droppers, depending on whether or not the malware code is contained in the dropper. When the malicious code is not contained within the dropper, it will be downloaded to the victim’s device from an external source. Some security researchers may call this two-stage type of dropper a downloader while referring to a two-stage dropper as the one that requires further steps to put different pieces of code together (by decompressing or executing different pieces of code) to build a final piece of malware.

The Command and Control (C2) is an attacker-controlled computer server that’s used to send commands to the malware running in the victim’s systems. It’s the way the malware communicates with its “owner.” There are multiple ways that a C2 can be established and, depending on the malware’s capabilities, the complexity of the commands and the communication that can be established may vary. For example, threat actors have been seen using cloud-based services, emails, blog comments, GitHub repositories, and DNS queries, among other things, for C2 communication.

There are different types of malware according to their capabilities, and sometimes, one malware piece can be classified as more than one type. The following is a list of the most common ones:

Worm: An autonomous program capable of replicating and propagating itself through the network.
Trojan: A program that appears to serve a designated purpose, but also has a hidden malicious capability to bypass security mechanisms, thus abusing the authorization that’s been given to it.
Rootkit: A set of software tools with administrator privileges, designed to hide the presence of other tools and hide their activities.
Ransomware: A computer program designed to deny access to a system or its information until a ransom has been paid.
Keylogger: Software or hardware that records keyboard events without the user’s knowledge.
Adware: Malware that offers user-specific advertising.
Spyware: Software that has been installed onto a system without the knowledge of the owner or the user, with the intention of gathering information about him/her and monitoring his/her activity.
Scareware: Malware that tricks computer users into visiting compromised websites.
Backdoor: The method by which someone can obtain administrator user access in a computer system, a network, or a software application.
Wiper: Malware that erases the hard drive of the computer it infects.
Exploit kit: A package that’s used to manage a collection of exploits that could use malware as a payload. When a victim visits a compromised website, it evaluates the vulnerabilities in the victim’s system in order to exploit certain vulnerabilities.

A malware family references a group of malicious software with common characteristics and, most likely, the same author. Sometimes, a malware family can be directly related to a specific threat actor. Sometimes, malware (or tool) is shared among different groups. This happens a lot with open source malware tools that are publicly available. Leveraging them helps the adversary disguise its identity.

Now let’s take a quick look at how we can collect data around malware pieces.

Using public sources for collection — OSINT

Open Source Intelligence (OSINT) is the process of collecting publicly available data. The most common sources that come to mind when talking about OSINT are social media, blogs, news, and the dark web. Essentially, any data that’s made publicly available can be used for OSINT purposes.

Important note

There are many great resources for someone looking to start collecting information: VirusTotal (https://www.virustotal.com/), CCSS Forum (https://www.ccssforum.org/), and URLHaus (https://urlhaus.abuse.ch/) are great places to get started with the collection process.

Also, take a look at OSINTCurio.us (https://osintcurio.us/) to learn more about OSINT resources and techniques.

Honeypots

A honeypot is a decoy system that imitates possible targets of attacks. A honeypot can be set up to detect, deflect, or counteract an attacker. All traffic that’s received is considered malicious and every interaction with the honeypot can be used to study the attacker’s techniques.

There are many types of honeypots (an interesting list can be found here: https://hack2interesting.com/honeypots-lets-collect-it-all/), but they are mostly divided into three categories: low interaction, medium interaction, and high interaction.

Low interaction honeypots simulate the transport layer and provide very limited access to the operating system. Medium interaction honeypots simulate the application layer in order to lure the attacker into sending the payload. Finally, high interaction honeypots usually involve real operating systems and applications. These ones are better for uncovering the abuse of unknown vulnerabilities.

Malware analysis and sandboxing

Malware analysis is the process of studying the functionality of malicious software. Typically, we can distinguish between two types of malware analysis: dynamic and static.

Static malware analysis refers to analyzing the software that’s used without executing it. Reverse engineering or reversing is a form of static malware analysis and is performed using a disassembler such as IDA or the more recent NSA tool, Ghidra, among others.

Dynamic malware analysis is performed by observing the behavior of the malware piece once it’s been executed. This type of analysis is usually performed in a controlled environment to avoid infecting production systems.

In the context of malware analysis, a sandbox is an isolated and controlled environment used to dynamically analyze malware pieces automatically. In a sandbox, the suspected malware piece is executed and its behavior is recorded.

Of course, things are not always this simple, and malware developers implement techniques to prevent the malware from being sandboxed. At the same time, security researchers develop their own techniques to bypass the threat actor’s anti-sandbox techniques. Despite this chase of cat and mouse, sandboxing systems are still a crucial part of the malware analysis process.

Tip

There are some great online sandboxing solutions, such as Any Run (https://any.run) and Hybrid Analysis (https://www.hybrid-analysis.com/). Cuckoo Sandbox (https://cuckoosandbox.org/) is an open source and offline sandboxing system for Windows, Linux, macOS, and Android.

Processing and exploitation

Once the data has been collected, it must be processed and exploited so that it can be converted into intelligence. The IOCs must be provided with context, and their relevance and reliability must be assessed.

One way to approach this is to break data into buckets and take advantage of the available frameworks in order to look for patterns.

We are going to quickly review three of the most commonly used intelligence frameworks: The Cyber Kill Chain®, The Diamond Model, and The MITRE ATT&CK™ Framework.

The Cyber Kill Chain®

Developed by Lockheed Martin, the Cyber Kill Chain® is a means to identify the steps the threat actor should follow in order to achieve their objective.

There are seven different steps:

1. Reconnaissance: Getting to know the victim using non-invasive techniques.

2. Weaponization: Generating the malicious payload that is going to be delivered.

3. Delivery: Delivering the weaponized artifact.

4. Exploitation: Achieving code execution on the victim’s system through the exploitation of a vulnerability.

5. Installation: Installing the final malware piece.

6. Command and Control (C2): Establishing a channel to communicate with the malware on the victim’s system.

7. Actions on objectives: With full access and communication, the attacker achieves their goal.

This model has been criticized for not being good enough to describe the way some modern attacks work, but at the same time, it has been praised for delimiting the points in which an attack can be stopped:

Figure 6 - Lockheed’s Martin Cyber Kill Chain®

The Diamond Model

The Diamond Model provides us with a simple way to track breach intrusions since it helps us establish the atomic elements involved in them. It comprises four main features: adversary, infrastructure, capability, and victim. These features are connected by the sociopolitical and technical axes:

Figure 7 - The Diamond Model

We will now have a look at the MITRE ATT&CK™ Framework.

MITRE ATT&CK™ Framework

The MITRE ATT&CK™ Framework is a descriptive model used to label and study the activities that a threat actor is capable of carrying out in order to get a foothold and operate inside an enterprise environment, a cloud environment, smartphones, or even industrial control systems.

The magic behind the ATT&CK™ Framework is that it provides a common taxonomy for the cybersecurity community to describe the adversary’s behavior. It works as a common language that both offensive and defensive researchers can use to better understand each other and to better communicate with people not specialized in the field.

On top of that, you not only can use it as you see fit, but you can also build on top of it, creating your own set of tactics, techniques, and procedures (TTPs).

12 tactics are used to encompass different sets of techniques. Each tactic represents a tactical goal; that is, the reason why the threat actor is showing a specific behavior. Each of these tactics is composed of a set of techniques and sub-techniques that describe specific threat actor behaviors.

The procedure is the specific way in which a threat actor implements a specific technique or sub-technique. One procedure can be expanded into multiple techniques and sub-techniques:

Figure 8 - The Enterprise Matrix

We will now have a look at bias and analysis.

Bias and analysis

Once all the necessary information has been processed, it is time to make sense of it; that is, search for the security issues and deliver this intelligence to the different strategic levels meeting the intelligence requirements that were identified during the planning step.

A lot has been written about how intelligence analysis should be done, especially in excellent books such as Structured Analytic Techniques for Intelligence Analysis (Heuer and Pherson, 2014), Critical Thinking for Strategic Intelligence (Pherson and Pherson, 2016), and Psychology of Intelligence Analysis (Heuer, 1999), among many others. These books employ many metaphors to describe the process of intelligence analysis.

My personal favorite is the one that compares the art of intelligence analysis with the art of mosaics: intelligence analysis is like trying to put the pieces of a mosaic together in which the pattern is not clear and the pieces continue to change in size, shape, and color.

One thing that an intelligence analyst cannot forget is that part of the practice is to challenge their own preconceptions and prejudices ceaselessly. Avoid confirmation bias, not to merely transmit the collected data, but to not fall for mirror imaging, clientelism, layering, and linear thinking. You should never influence the analysis so that it suits your needs or views. There are many techniques that can be used to mitigate analyst bias.

Some common traits are used to define a good intelligence analyst: he or she must have specific knowledge in more than one field; he or she must have a good spoken and written expression; and, most important of all, he or she must have the ability to synthesize the background of a situation almost intuitively.

In conclusion, we can close with the asseveration that in order to generate effective and relevant intelligence, there has to be a continuous intelligence process in place, with information from both internal and external sources being continually collected, processed, and analyzed.

This analysis must be tackled from different angles and by people with different perspectives and backgrounds in order to minimize the risk of falling into our own cognitive biases.

In addition, establishing good mechanisms for both disseminating quality and relevant intelligence reports, as well as getting feedback from the recipients, is key to enriching and improving this process.

Summary

In this overview, we’ve covered the definitions of cyber threat intelligence (CTI) and advanced persistent threats (APTs). We reviewed each of the steps involved in the intelligence cycle and provided an overview of how to carry out data collection and processing. Finally, we examined one of the main challenges that intelligence analysts face: analyst bias.

This introduction to cyber threat intelligence is part of Valentina Palacin’s Practical Threat Intelligence and Data-Drive Threat Hunting book. If you want to start out in the cyber intelligence and threat hunting domains, this guide will help you explore the ATT&CK™ Framework and open source tools to implement a threat hunting program from scratch.

Configuring the Salesforce Mobile App

Packt — Tue, 13 Apr 2021 10:56:41 GMT

This guide will show you how to configure the Salesforce Mobile App using declarative customization and actions in the Salesforce Lightning platform. You will learn about the features available to customize the Salesforce mobile application user interface, and you will learn how to configure actions and action layouts and optimize the Salesforce Mobile App user experience for users accessing the Lightning Platform on the go.

We will cover the following topics:

Exam objectives — Mobile apps
Installing the Salesforce Mobile App
Configuring the Salesforce mobile application user interface
Building global and object-specific actions and action layouts

Exam objectives — Mobile apps

To complete the Mobile section of the Certified Platform App Builder exam, app builders are expected to be able to carry out the following tasks:

Describe the declarative customization options available for the Salesforce mobile application user interface.
Given a set of requirements, determine the appropriate global and object-specific actions and action layouts to optimize the Salesforce mobile application UX (user experience).

Reference - Salesforce Certified Platform App Builder Exam Guide

This guide is published by Salesforce and can be referenced at https://trailhead.salesforce.com/help?article=Salesforce-Certified-Platform-App-Builder-Exam-Guide.

In the Salesforce Certified Platform App Builder Exam Guide, the total number of questions is given, along with a percentage breakdown for each of the objectives, and an indication of the number of features/functions that can be expected in each of the objectives.

By analyzing these objectives, percentages, and question counts, we can determine the likely number of questions that will appear in the exam. For the Mobile objective, there are likely to be three questions in total (5% of 60 total exam questions).

To help reinforce the skills and knowledge required to create reports and build dashboards, there is some practical work for you to do in Salesforce, so if you do not have a Salesforce environment in which to carry this out, first create a free developer org.

The options and capabilities of the mobile features will be covered in this guide, and you will learn how mobile features enable you to design an appropriate user interface on the mobile platform and optimize the user experience.

Let’s now look at the Salesforce Mobile App and learn how to set up the app on mobile devices.

Installing the Salesforce Mobile App

Mobile phones and devices are ubiquitous within both our private lives and the business environment. In the current climate of mixing work from the office, the home, remotely, and on the move, organizations of all sizes have realized the benefits of using mobile devices to access business applications in today’s mobile-first world.

Mobile application development, however, is not always cost-effective, as the final product can be expensive to build or fails to meet user requirements. Not all mobile apps developed pass the test for usability, with apps built for mobile devices often lacking functionality and providing a poor user experience. New mobile apps or migrations of desktop apps to mobile sometimes fail as they are simply mobile screen overlays that are not fit for purpose. This can result in poor user adoption.

The Salesforce Mobile App differs in its approach as it is built within the Lightning Platform and configurations carried in the desktop are fully integrated and reflected in the mobile app. The mobile offering provided with the Lightning Platform provides a common security model and a common set of configuration features that provide an innovative, responsive, and easy-to-use mobile experience for users.

The Salesforce mobile web browser is no longer available

Prior to the Summer 2020 release, there were two options for accessing Salesforce mobile features using a mobile device. The first was the downloadable Salesforce Mobile App and then there was a second option, the Salesforce mobile web browser app, which did not require any software installation and used the web browser on mobile devices.

Starting from the Summer 2020 release, the Salesforce mobile web browser app is no longer available. Users must use the downloadable Salesforce Mobile App to access the mobile features on mobile devices, as detailed in the following article: https://help.salesforce.com/articleView?id=000349471&language=en_US&type=1&mode=1

The Salesforce Mobile App includes many of the customizations that have been configured within the desktop version of your organization and are therefore already partly tailored out of the box to deliver your mobile application requirements.

Let’s now look at which mobile devices are supported for use with the Salesforce Mobile App.

Supported mobile devices for the Salesforce Mobile App

The Salesforce Mobile App is accessible from most Salesforce editions and types of user licenses and is supported for use with the following two operating systems and versions: Apple iOS version 12.0 or later, and Android version 7.0 or later.

Salesforce conducts ongoing tests using various mobile phones and tablets to verify that the Salesforce Mobile App operates correctly with the stated platform.

At the time of writing, Salesforce has identified the following mobile phones as supported for use with the Apple iOS platform:

iPhone 11 Pro and Pro Max
iPhone 11
iPhone XR
iPhone XS and XS Max
iPhone X
iPhone 8 and 8 Plus
iPhone 7 and 7 Plus
iPhone SE
iPhone 6S
iPhone 6

Salesforce has identified the following tablets as being supported for use with the Apple iOS platform:

iPad Pro 10.5 inch
iPad Air 2
iPad Mini 4

At the time of writing, Salesforce has identified the following mobile phones as being supported for use with the Android platform:

Google Pixel 4 XL
Google Pixel 3
Samsung Galaxy S10 and S10+
Samsung Galaxy S9 and S9+
Samsung Galaxy S8 and S8+
Samsung Galaxy S7
Samsung Galaxy Note 10+
Samsung Galaxy Note 9
Samsung Galaxy Note 8

Salesforce Mobile App-supported devices and minimum platform requirements

Salesforce may, at any time, change the list of supported devices and minimum platform requirements for the Salesforce Mobile App. The latest set of supported devices and minimum platform requirements can be accessed from the following article:

https://help.salesforce.com/articleView?id=salesforce_app_requirements.htm&type=5

We will now look at how to set up the Salesforce Mobile App on a supported mobile device.

Setting up the Salesforce Mobile App

The Salesforce Mobile App is available from the App Store for mobile devices that are running on the iOS platform and from Google Play for mobile devices that are running on the Android platform.

We will step through the process of setting up the Salesforce Mobile App on an iOS device. The set-up process is similar for an Android device, although there may be some minor variations.

To access the installation links for the Salesforce Mobile App on an iOS device, perform the following steps:

Navigate to the Getting started with the Salesforce Mobile App page by browsing to the web URL located at https://www.salesforce.com/solutions/mobile/getting-started/, as shown in the following screenshot:

Figure 1 - Getting Started

In this setup, we will install the Salesforce Mobile App on an Apple iPhone that runs under the iOS platform, so here we’ll select the App Store option.

2. Click on Download on the App Store within the Getting Started screen to navigate to the Salesforce App Store menu, as shown in the following screenshot:

Figure 2 - Salesforce on the App Store

3. Click on Install in the App Store.

4. Wait for a few moments for the installation to complete and the screen will then change to present an option to open the Salesforce Mobile App, as shown in the following screenshot:

Figure 3 - Salesforce Mobile App open

5. Click on Open in the Salesforce Mobile App and this action will (upon navigation to the Salesforce Mobile App for the first time) present the Allow Access screen, as shown in the following screenshot:

Figure 4 - Salesforce Mobile App access permissions

6. Click on Allow to permit the Salesforce Mobile App to gain access to your Salesforce organization, whereby the screen will then change to the Welcome to the Salesforce Mobile App screen, as shown in the following screenshot:

Figure 5 - The Salesforce Mobile App welcome screen

7. Finally, click on Get Started to navigate to the Salesforce Mobile App menu, as shown in the following screenshot:

Figure 6 - The Salesforce Mobile App menu

More information about setting up the Salesforce Mobile App

To find out more information about setting up the Salesforce Mobile App, refer to the following article:

https://help.salesforce.com/articleView?id=salesforce_app_set_up.htm&type=5

This will set up the mobile app. Now let’s look at the mobile app UI (user interface) options in the Lightning Platform.

Configuring the Salesforce mobile application user interface

The options and capabilities of the Salesforce Mobile App are not the same as the full Lightning Platform application, and there are many differences within the Salesforce Mobile App, in particular, concerning the ways in which data is entered, screens are presented, and navigation is carried out between screens.

Salesforce has built mobile-friendly alternatives to the way information is captured, and navigation takes place accordingly to cater for the much smaller screen real estate and the reduced capability of a mobile device to display functions and features to a user.

Differences between the full Lightning Experience app and the Salesforce Mobile App

For more detailed information about the features that are not available in the Salesforce Mobile App when compared to the full Lightning Experience app, refer to the following article:

https://help.salesforce.com/articleView?id=limits_mobile_sf1_parent.htm&type=5

The features and capabilities that are available for configuring the Salesforce Mobile App user interface are as follows:

Lightning App Navigation Menu: The Lightning app navigation menu can be changed by users who have permission to personalize the desktop navigation bar. This also allows a user who has permission to edit the desktop navigation bar, for a given app, to reorder items in the navigation menu within the Salesforce Mobile App, for that particular app.
Mobile Only Default Navigation Menu: The Mobile Only app allows you to create navigation items that become the default set of items within the Salesforce Mobile App.
Mobile App Branding: This provides the facilities to configure the way in which the app appears.
Global Quick Actions: These actions in the Lightning Platform can be thought of as shortcuts that appear within the user interface. A given global quick action is available in both the desktop and mobile apps and can be used to create records, log calls, and so on.
Object-specific Quick Actions: In the same way as global quick actions, object-specific quick actions can be thought of as shortcuts that appear within the user interface. They are also available in both the desktop and mobile apps, but are associated with a specific Lightning Platform object type and can be used to update as well as create records, log calls, and so on.

Customization options in the Salesforce Mobile App

For more information about customizing the Salesforce Mobile App, refer to the following article:

https://help.salesforce.com/articleView?id=salesforce_app_customize.htm&type=5

Having looked at options for configuring the user interface, let’s now look at how the changes are reflected when personalizing the Lightning app menu in the Salesforce Mobile App.

Customizing a Lightning app navigation menu in the Salesforce mobile app

The Lightning app navigation menu can be personalized by any user who has permission to modify the navigation bar whereby they are able to add, remove (only for items that they have added), rename, and reorder items.

The modifications to the navigation bar are also applied to the navigation menu and navigation bar for the specific Lightning app within the Salesforce Mobile App, as shown in the following screenshot:

Figure 7 - The Lightning app navigation menu

The preceding screenshot demonstrates reordering by placing the Leads tab to appear as the first item that is reflected in the Salesforce mobile app, as shown in the following screenshot:

Figure 8 - The Lightning app navigation menu in mobile

Having identified how the Lightning app navigation menu can be personalized, let’s now look at the Mobile Only default navigation menu in the Salesforce mobile app.

Customizing the Mobile Only default navigation menu in the Salesforce Mobile App

The Mobile Only default navigation menu can only be configured by users with system administrator permissions and is used to create navigation items that become the default set of items within the Salesforce mobile app.

By configuring the items that are presented to users, you can include Lightning pages, Visualforce pages, Lightning components, and so on in the default navigation menu and navigation bar of the Salesforce mobile app.

Modification of the navigation bar for the Mobile Only default navigation menu can be carried out by performing the following steps:

Navigate to SETUP and then search for Salesforce Navigation in the Quick Find search box located at the top of the Setup menu on the left sidebar.
Click on Salesforce Navigation in the SETUP menu.
In the Salesforce Navigation setup screen, choose the items from the Available list and move them to the Selected list, as shown in the following screenshot:

Figure 9 - The Mobile Only default navigation menu

4. Finally, click on Save.

The Mobile Only default option shows the selected items in the Salesforce mobile app, as shown in the following screenshot:

Figure 10 - The Mobile Only default navigation menu in mobile

Customization options in the Mobile Only default navigation menu

For more information about customizing the Mobile Only default navigation menu, refer to the following article:

https://help.salesforce.com/articleView?id=salesforce_app_customize.htm&type=5

We will now look at the mechanism for setting branding for the Salesforce mobile app.

Mobile app branding

Mobile app branding provides the facilities to configure the way in which the app appears. Here, the features that can be configured are the loading page logo, the loading background color, and the header background color.

Mobile app branding modifications can be carried out by performing the following steps:

Navigate to SETUP and then search for Branding in the Quick Find search box located at the top of the SETUP menu on the left sidebar.
Click on Salesforce Branding in the SETUP menu.
In the Salesforce Branding setup screen, click on Edit.
In the Salesforce Branding edit page, choose the colors for Brand Color and Loading Page Color, and then choose a Loading Page Logo, as shown in the following screenshot:

Figure 11 - Mobile app branding

The loading page color and loading page logo appear within the Salesforce mobile app, as shown in the following screenshot:

Figure 12 - Mobile app branding on a mobile device

The brand color appears within the Salesforce mobile app, as shown in the following screenshot:

Figure 13 - Mobile app branding in a mobile device

Next, we’ll look at the mechanism for building global and object-specific actions and action layouts in the Lightning Platform.

Building global and object-specific actions and action layouts

Global and object-specific quick actions in the Lightning Platform can be thought of as shortcuts that appear within the user interface. They are two forms of what is known as a quick action, which is available in both desktop and mobile apps and can be used to create records, update records, log calls, and so on.

In the Salesforce mobile app, global and object-specific quick actions offer a quick way to access functions that launch a specific action or custom process automation that exists in the Lightning Platform.

In addition to the custom actions that are tailored to your business processes, the Salesforce Mobile App comes with standard pre-built actions. The quick actions are located within the action bar and the action menu that appears at the top of the screen of the mobile device.

Quick actions that are used to capture data input have their own type of page layout called an action layout and allow you to reduce the number of fields to ensure that only those that are absolutely essential are presented to the mobile user.

In addition to presenting a limited set of fields for data capture, you can also set fields with given values, which are shown on the page layout, to save the user time when there are records that have common values that are to be entered. This feature is called prepopulate fields and is available in both the desktop and the Salesforce Mobile Apps.

As an app builder, you will need to understand how to build global and object-specific actions and action layouts in the Lightning Platform and understand the following key differences.

Global quick actions can be used to create records that are not associated with any other record (that is, there is no option to set an object relationship field). They are called global quick actions because they can be placed anywhere within the Lightning Platform where actions are supported such as the Home page, the Chatter tab, object pages, and custom Lightning app pages.

Global quick actions

For more information about the actions that can be carried out using global quick actions, refer to the following article:

https://help.salesforce.com/articleView?id=actions_overview_global.htm&type=5

Object-specific actions, on the other hand, can be used to create and also update records as they operate in the context of a given Lightning Platform object type. Within the Salesforce mobile app, object-specific quick actions can be placed on record detail pages. So, for example, an action associated with the lead object is only available when viewing a lead record.

Object-specific actions

For more information about the actions that can be carried out using object-specific actions, refer to the following article:

https://help.salesforce.com/articleView?id=actions_overview_object_specific.htm&type=5

Let’s look at how to create global quick actions in the Lightning Platform.

Creating global quick actions

In this example, we will add a New Lead action to global layouts that will allow mobile users to create lead record details from within a Chatter thread. Let’s go ahead and configure a global quick action by performing the following steps:

Navigate to SETUP and then search for Actions in the Quick Find search box located at the top of the SETUP menu on the left sidebar.
Click on Global Actions in the SETUP menu.
Click on New Action on the Global Actions setup page.
In the Enter Action Information dialog screen, enter the following:

Action Type: Create a Record
Target Object: Lead
Standard Label Type: — None —
Label: New Lead (Global Action)
Name: New_Lead_Global_Action
Description: Platform App Builder Certification Guide, Custom Lead Assignment Scenario, Global Action New Lead
Create Feed Item: Enabled (when enabled, a feed item is created along with the action. The compact layout for the target object sets the fields for the feed item.)
Success Message: Lead Created
Icon: Accept the default lead icon for the Lead (icons change according to the target object.)

5. Finally, click on the Save button to save the global action, as shown in the following screenshot:

Figure 14 - Global actions

The options for the action types are as follows:

Create a Record
Send Email
Log a Call
Custom Visualforce
Custom Canvas
Lightning Component

Upon saving the global action, the layout setup screen appears as shown in the following screenshot:

Figure 15 - Global actions layout

The following fields are added as default: Name, Email, Phone, Company, Title, and Lead Status. The Name, Company and Lead Status fields marked with a red asterisk are mandatory fields and cannot be removed from the page layout.

6. Remove the following fields, Phone, and Title, by clicking on the Remove icon, as shown in the following screenshot:

Figure 16 - Global actions - removing layout

7. Add the following fields by dragging and dropping the field section, as shown in the following screenshot:

Figure 17 - Global actions - adding layout

8. Finally, click on the Save button to save the global action layout.

Limiting the number of fields on an action layout

Salesforce recommends including fewer than five and observing a maximum number of eight fields on an action layout, as referenced in the following article:

https://trailhead.salesforce.com/en/content/learn/modules/salesforce1_mobile_app/salesforce1_mobile_app_actions_global

To present the New Lead global action as an option in the user interface, we now need to add the global action to a publisher layout. We can create custom publisher layouts. However, there is a standard layout that we will use called the Global Publisher layout. You may need to check that this layout is included within the profiles within the Publisher Layout Assignment if you have already made any changes to the Global Publisher layout.

Let’s now go ahead and add the global action to the global publisher layout by performing the following steps:

Navigate to SETUP and then search for Publisher in the Quick Find search box located at the top of the SETUP menu on the left sidebar.
Click on Publisher Layouts in the SETUP menu.
Click on Edit for the standard Global Publisher layout.
If you are editing Global Publisher for mobile actions for the first time, you will need to click the override the predefined actions link, as shown in the following screenshot:

Figure 18 - Global Publisher override

5. Optionally, click on override the predefined actions in the Global Publisher layout.

6. Ensure that New Lead (Global Action) is included within Salesforce Mobile and Lightning Experience Actions, as shown in the following screenshot:

Figure 19 - Global Publisher layout

7. Finally, click on the Save button to save the publisher layout.

Let’s now see how this looks in the Salesforce Mobile App by navigating to the Chatter item (this can be found in the Mobile Only app). The New Lead (Global Action) option will be presented, as shown in the following screenshot:

Figure 20 - Global Publisher layout mobile

Next, we’ll turn to the practicalities of building reports that satisfy a specific business requirement by implementing a report with the use of the Lightning report builder.

Configuring object-specific actions

In this example, we will add a Deactivate action to the location records that will allow mobile users to easily mark the location as inactive from within a location record.

Let’s go ahead and configure an object-specific quick action by performing the following steps:

Navigate to SETUP and then search for Object Manager in the Quick Find search box located at the top of the SETUP menu on the left sidebar.
Click on Object Manager in the SETUP menu.
In the Object Manager setup screen, search for the Location object.
Click on Location in Object Manager in the SETUP menu.
In the Location object manager setup screen, click on Buttons, Links, and Actions from within the Location object setup menu on the left sidebar.
In the Buttons, Links, and Actions section, click on New Action.
In the Enter Action Information dialog screen, enter the following information:

Action Type: Update a Record
Standard Label Type: — None —
Label: Deactivate Location
Name: Deactivate_Location
Description: Platform App Builder Certification Guide, Custom Lead Assignment Scenario, Object-specific Action Deactivate Location
Success Message: Location Deactivated
Icon: Accept the default icon

8. Finally, click on the Save button to save the global action, as shown in the following screenshot:

Figure 21 - Object-specific actions

The options for Action Types are as follows:

Create a Record
Send Email
Log a Call
Custom Visualforce
Update a Record
Lightning Component
Flow

Upon saving the global action, the layout setup screen appears as shown in the following screenshot:

Figure 22 - Object-specific actions layout

The following field, Location Name, is added by default and is marked with a red asterisk to show it is a mandatory field and cannot be removed from the page layout.

Add the Active field to the action layout.
Click on the Save button to save the action layout.
Upon saving the action layout, click on New in the Predefined Field Values section, as shown in the following screenshot:

Figure 23 - Object-specific action predefined

4. In the Enter Action Information dialog screen, enter the following information:

Field Name: Active
Specify New Field Value: Formula Value (Checkbox): FALSE

5. Click on the Save button to save the Predefined Field Value, as shown in the following screenshot:

Figure 24 - Object-specific predefined action saved

Let’s now add the object-specific action to the Location page layout by performing the following steps:

6. In the Location object manager setup screen, click on Page Layout from within the Location object setup menu on the left sidebar.

7. Click on the Location Layout page layout.

8. Drag the Deactivate Location action to the Salesforce Mobile and Lightning Experience Actions section of the page layout, as shown in the following screenshot:

Figure 25 - Object-specific action page layout

9. Finally, click on the Save button to save the page layout.

Let’s now see how this looks in the Salesforce Mobile App by navigating to the Chatter item (this can be found in the Mobile Only app). The New Lead (Global Action) option will be presented, as shown in the following screenshot:

Figure 26 - Object-specific action - mobile

Summary

In this guide for Salesforce developers, we have looked at the options for configuring mobile using declarative customization and actions in the Salesforce Lightning Platform.

You have learned about the setup and installation of the Salesforce Mobile App and about options for customizing the Salesforce Mobile App user interface.

You have learned how to configure quick actions within the Salesforce Mobile App that can be used to implement global quick actions, object-specific actions, and action layouts to optimize the Salesforce mobile application user experience.

With this knowledge, you are now well-equipped to tackle the Mobile section of the Salesforce Certified Platform App Builder exam.

This article is part of the Salesforce Platform App Builder Certification Guide by Paul Goodey. To learn how to build apps on the Salesforce Platform and pass the Salesforce Platform App Builder certification exam, check out the complete book here.

Stories by Packt on Medium

Top 5 Tech Reads from the Packt Christmas Sale

How to use Semaphores in POSIX Concurrency Control

Concept of the semaphore and its counterpart object in the pthread library: the POSIX semaphore, with an example.

POSIX Semaphores

General semaphores

Summary:

About the author:

Role of Declaration Files in the Implementation of TypeScript

In this article, we will look at how we can enhance a JavaScript library and add syntactic sugar through declaration files.

Declaration files

Declaration file typing

Function overloading

Nested namespaces

Classes

Static properties and functions

Abstract classes

Generics

Conditional types

Conditional type inference

Declaration file summary

About the author

Essential Statistics for Data Assessment

Classifying numerical and categorical variables

Distinguishing between numerical and categorical variables

Understanding mean, median, and mode

Mean

Median

Learning about variance, standard deviation, quartiles, percentiles, and skewness

Standard deviation

Quartiles

Skewness

Revisiting outlier detection

Knowing how to handle categorical variables and mixed data types

Knowing how to handle categorical variables and mixed data types

Frequencies and proportions

Transforming a continuous variable to a categorical one

Using bivariate and multivariate descriptive statistics

Covariance

Cross-tabulation

In conclusion

Implementing Terminal I/O in Rust

Technical requirements

Introducing terminal I/O fundamentals

Characteristics of terminals

The Termion crate

What will we build?

Working with the terminal UI (size, color, styles) and cursors

Writing data structures and the main() function

Initializing the text viewer and getting the terminal size

Displaying a document and styling the terminal color, styles, and cursor position

Exiting the text viewer

Processing keyboard inputs and scrolling

Listening to keystrokes from the user

Positioning the terminal cursor

Enabling scrolling on the terminal

Processing mouse inputs

Summary

Upgrading Existing .NET Apps to .NET 5

Upgrading Existing .NET Apps to .NET 5

Choosing the migration approach

Lift and shift versus architectural transformation

Replacing third-party libraries with platform-provided features

Points to ponder regarding an as-is upgrade

Unsupported technologies

Unsupported Windows OS-specific features

Transforming application configuration

Choosing the Entity Framework version

Using .NET Framework compatibility

Upgrading third-party NuGet packages

Summary

Further reading

Exploring SharePoint Migration

Getting ready: know your content

Know your customers

Migration — choosing your weapon of choice

Content database migration

Using a migration tool

PowerShell migration

Manual migrations