texts: Hidden Markov Model: Forward Algorithm implementation in Python

texts:
I am learning Hidden Markov Model and its implementation for Stock Price Prediction. I am trying to implement the Forward Algorithm according to this paper.

Here I found an implementation of the Forward Algorithm in Python.

import pandas as pd
import numpy as np

V = np.array([0, 1, 1, 2, 0, 1, 2, 0, 1, 0, 2])

# Transition Probabilities
a = np.array(((0.54, 0.46), (0.49, 0.51)))

# Emission Probabilities
b = np.array(((0.16, 0.26, 0.58), (0.25, 0.28, 0.47)))

# # Equal Probabilities for the initial distribution
pi = np.array((0.5, 0.5))

def forward(V, a, b, pi):
    alpha = np.zeros((V.shape[0], a.shape[0]))
    alpha[0, :] = initial_distribution * b[:, V[0]]

    for t in range(1, V.shape[0]):
        for j in range(a.shape[0]):
            alpha[t, j] = alpha[t - 1].dot(a[:, j]) * b[j, V[t]]

    return alpha

alpha = forward(V, a, b, pi)

But it seems to me that it does not include (c) and (d) steps from the algorithm. So I added them:

def forward(V, a, b, pi):
    p = 1
    alpha = np.zeros((V.shape[0], a.shape[0]))
    alpha[0, :] = pi * b[:, V[0]]

    for t in range(1, V.shape[0]):
        probability_of_observation = 0 #my code
        for j in range(a.shape[0]):
            alpha[t, j] = alpha[t - 1].dot(a[:, j]) * b[j, V[t]]
            probability_of_observation += alpha[t, j]  #my code
        p = p * probability_of_observation #my code

    return p #changed

p = forward(V, a, b, pi)  #changed

Does my code coincide with the given algorithm?


texts: How do I read the cord_19_embeddings_2020-07-16.csv from the COVID-19 Open Research Dataset Challenge (CORD-19) on Kaggle?

texts:
What I'm trying to do
I wanted to use the CORD19 word embeddings csv to map them to certain findings from the rest of the dataset, but as we can see there are no stings in the first column.
The way I know word or sentence embeddings, is what they map a word or a sentence to multiple vectors.
The values in the first column look somewhat like hashes, and they are the main problem I can't use the dataset.
Can somebody give me a pointer on what I'm looking at and how to use them?
I have not found documentation, usage examples or submissions on kaggle that have explained or outlined how this file is supposed to be used.


texts: Unable to understand which features to choose

texts:
I am a newbie here, but I am trying to work with a dataset which gives the attempt at the goal by a footballer,which will predict one of 2 possible outputs - whether or not they could score the goal or not.
I have done some basic cleaning but I am still getting only 60% accuracy on whatever classifier I use from sklearn.
I have removed a few features which I thought would not contribute to the 'y' value, and encoded a few values, but still it doesn't bump up the accuracy.
I am not sure how to attach a csv, but this is what the data looks like -- data
For the rows that 'is_goal' doesn't have a value, the aim is to find out if a goal was scored or not.
Should I attach the notebook as well to show what work I've done till now?


texts: How does Api.ai Google dialogueflow classifies "intent" and extracts data from slots

texts:
I am trying to build a very naive version of Api.ai, now Google DailogueFlow.
I wanted to know two things.

How DF classifies sentences with entities in it that can be user created and/or things like date, place, time, currency or even random text (which they call 'system.any')
How data is extracted from slots after user map the slots with the sentence.

What algorithm/libraries and techniques do they use to classify, extract and identify user queries.


texts: Finding the wiggles pattern in the original dataset. (Wiggles appear after performing division by another dataset)

texts: I have multiple measurements regarding scientific observations. The problem is that there is a subtle noise pattern caused by the instrument - the wiggles. These wiggles are invisible when looking at the scientific data itself (plotting x vs y), but become clear when I divide one measurement by another taken at a later time (when the wiggle pattern between them changed/shifted). My goal is to correct the "wiggles" for each individual measurement. So far I've tried dividing the measurement by a shifted copy (no success, the data has too many features it seems) and to divide by a smooth function of the data (a kind of polynomial normalization, but no wiggles were found after).


texts: Feature engineering from date, mean and standard deviation

texts:
I have a multi class classification problem where I should predict the passengers for flights (0-7 classes). The training set consists of the following features:

Date of the flight
Mean of the weeks that the passengers bought their tickets
Standard Deviation of the above

I extracted from the date, the day of the week, the month and if the flight is in a high season. What other features could I extract from the date? How could I use the mean and standard deviation to create new features?


texts: Python Script using pandas to plot histograms between the features

texts:
I am working around with data from Kaggle : Titanic Competition dataset.

Script

#cols : List of all column header
cols = list(df.columns.values)
#script to plot histograms of all columns with respect to column: "Survived"
for val in cols:
       df.groupby('Survived')[val].hist(alpha = 0.2)

Code of importance (COI)

df.groupby('Survived')['Age'].hist(alpha = 0.3)

Output

The COI works but the Script is not working. The script will help to plot all the histograms in one go without manually inserting the column in the COI.

The script is running indefinetly.
The COI also runs indefinetly but once I execute the following code:

df.groupby('Survived').Age.hist(alpha = 0.2) #This runs flawlessly

COI runs perfectly. (Not able to figure out why).

The scripts are written in Kaggel Kernel.

Given the above details, please help me figure out a way to achieve the above script either by making improvements to the Script or an alternate way of doing the same.


texts:
I want to predict the grade of a (german) health insurance by some features it offers and I am not sure about the best way to define these features.

How should I treat a variable which can be "80%", "90%" or "100%". If I use it categorical I loose the fact that 100% is better than 90%.
If numerical is better, should I use 80,90,100 or 0.8, 0.9, 1 ?
Categorical Variables like "multi bed", "double bed", "singe bed" also have a "is-better-than" relation but how should I transform them into numerical values? Ascending integers, or from -1 to 1 or from 0 to 1. Or does all this make no sense at all.
Should I treat grade as a numerical target or as categorical (1.0, 1.1 ...) as only these values are allowed outcomes.
What method of prediction do U suggest? I have around 80 features, most of the categorical (mostly yes/no).


texts: Relating changes of a value in time to known events

texts:
I work with two datasets. The first dataset contains fluor values measured every minute. The second dataset contains certain events and their time. We know that these events cause peaks in fluor values shortly before and shortly after the event time.

A simplified reproducible example in R:

Here I provide a simplified version of the R code I use to relate the fluor values to events. I have a series of fluor values measured every minute. Next I have a second dataset with three events A, B and C that occur at three different times. It seems there are three large peaks in fluor value around each event.

The objective is to classify each value as corresponding to an event whenever the time of that value falls within 15 minutes of that time of an event. Fluor values that fall outside of any events are categorized as NA.

Set seed

set.seed(10)

Generate fluor values dataset and events dataset

values <- data.table(date.time.value=seq(ymd_hms("2018-01-01 00:00:00"), by= "min", length.out = 200),fluor=5*sin(seq(0,20,length.out = 200)))+(abs(rnorm(200,6,1))+2)
events <- data.table(event=c("A","B","C"),date.time.event=c(ymd_hms("2018-01-01 00:20:00"),ymd_hms("2018-01-01 01:20:00"),ymd_hms("2018-01-01 02:20:00")))

Using the data.table package: I add an "event" variable to the values dataset. That variable takes on the name of an event (in this case A, B or C) whenever the date.time.value is within 15 minutes of a date.time.event.

values[, event:=events[.SD[, .(d_dn=date.time.value-15*60, d_up=date.time.value+15*60)], on=.(date.time.event>=d_dn, date.time.event<=d_up), event]]

Here's what the result looks like. We see that we have one fluor value peak for each event A, B and C and that the other fluor values have NA as event:

ggplot(values, aes(date.time.value, fluor,col=event)) + geom_point()

A real example:

I managed to relate the two datasets by coloring the fluor values that occur several minutes before or several minutes after an event. This is what the result looks like:

What I'm missing:

I would like to define "peaks" so that in my simplified example all the green-colored fluor values of event B belong to a specific "peak number x". Currently it's impossible to distinguish between fluor values of the same event but from different peaks because I only have observations of fluor with a corresponding variable "event".
In the simplified example you can see (exagerated) that quite a lot of high fluor values are considered to be NA even though their high value is most likely are caused by an event. Hence I'm looking for a better way to relate fluor values to events than simply saying "15 minutes before or 15 minutes after".

I'm almost certain that there must be some kind of R-package that's specifically built for my problem here but I just can't figure out which one...


texts: I got a overall mean average precision score of 0 for a recommendation engine

texts: I just wanted to know if receiving an overall MAP score of 0 in a recommendation engine was possible, or a sign that my calculation or my logic for the engine was wrong.


texts: How to preprocess different kinds of data (continuous, discrete, categorical) before Decision Tree learning

texts:
I want to use some Decision Tree learning, such as the Random Forest classifier.

I have data of different types: continuous, discrete and categorical. How do I have to preprocess data in order to have consistent results?


texts: 1st order Taylor Series derivative calculation for autoregressive model

texts:
I wrote a blog post where I calculated the Taylor Series of an autoregressive function. It is not strictly the Taylor Series, but some variant (I guess). I'm mostly concerned about whether the derivatives look okay. I noticed I made a mistake and fixed the issue. It seemed simple enough,but after finding an error, I started to doubt myself.
$$f(t+1) = w_{t+1} \cdot f(t) $$
$$y^{*}_{t+1} = f(t+1)-{\frac {f'(t+1)}{1!}}(-t-1+t)$$
$$y^{*}_{t+1} = w_{t+1} f(t) + \dfrac{d}{df(t)}w_{t+1}f(t) + \dfrac{d}{dw_{t+1}}w_{t+1}f(t)$$
$$y'_{t+1} = w_{t+1} f(t) + w_{t+1} + f(t)$$
The details can be found in the blog post:
EDIT 7/6/20:
The AR form:
$$y^{*}_{t+1}=c+\sum _{{I=0}}^{L}w _{t+1-i}y_{{t-i}}+\varepsilon _{t}$$
f(t) is a recursive dense layer, y is the predicted output, and w are the weights, and L are the number of lag components. For the simple case where the next value only depends on the previous value, I got the following result.
$$f(t+1) = w_{t+1} \cdot f(t) $$
$$y^{*}_{t+1} = f(t+1)-{\frac {f'(t+1)}{1!}}(-t-1+t)$$
$$y^{*}_{t+1} = w_{t+1} f(t) + \dfrac{d}{df(t)}w_{t+1}f(t)$$
$$y'_{t+1} = w_{t+1} f(t) + w_{t+1}$$
EDIT 7/7/20:
The function f(t) represents y(t) with an error term. The error term might have some random process, but I'm going to assume that the errors are independent.
$$f(t+1) = w_{t+1} \cdot y(t) + \epsilon_t$$
EDIT 7/9/20:
Changed the dimensionality of w_t+1 to w_t.
$$f(t+1) = w_{t} \cdot f(t) $$
$$y^{*}_{t+1} = f(t+1)-{\frac {f'(t+1)}{1!}}(-t-1+t)$$
$$y^{*}_{t+1} = w_{t} f(t) + \dfrac{d}{df(t)}w_{t}f(t)$$
$$y'_{t+1} = w_{t} f(t) + w_{t}$$


texts: Pre-trained models for classifying images with more than 3 input channels

texts: I am working on an image classification problem. My input images have 19 channels. I tried building a CNN model that can handle such inputs. Now I want to learn transfer learning along with this. But I am not able to use the models such as Inception and VGG as they are not trained for such inputs with more than 3 input channels. So, Is there any other pre-trained model which can work on such inputs? I hope there will be some models that were trained to work on satellite images/fMRI/video inputs. If there are, I will be able to make use of them.


texts:
I am reading some research papers about graph convolutional neural networks and I have seen the term "latent representations" used a lot. For instance, "the model was able to learn latent representations of the nodes of the graph".
What does the word "latent" means here? Is it the same as latent features?


texts: Splitting large multi class dataset using leave one out scheme into train and test

texts: I am doing some supervised learning using neural networks, and i have a Targets array containing 1906 samples, which contain 664 unique values. min. count of each unique value==2, by design. Is there a smarter way to split this dataset into train and test, using a leaveoneout scheme to pick randomly 1 sample from each class and put it in the test set and use the rest for training, before i get down to explicitly iterating over all my values? I am using python, numpy, sklearn and pytorch btw!


texts: Determine if a piece of text is a job ad

texts:
I am interested in learning more about the field of ML and more specifically in text classification.

Given a piece of text, is there any technique or algorithm to determine if that piece of text refers to a job ad or not?

Thank you.


texts:
I have a couple of questions and I was wondering if you could answer them.

I have a bunch of images of the cars (side view only). I would like to train a model with those images. My objects of interest are 3 types of trucks that have different trailers. I rarely see two target objects at once in an image (maybe $1\over2$ in every 1000 images). However, I do see other types of cars that I do not want to detect.

My questions are:

Do you think I should tackle this problem as a detection task or classification task? (For example, should I consider multilabel classification or omit those pictures).

Should I also include other vehicles that I do not want to detect in my training dataset? Let's say that I do not assign a bounding box to them, but include them in training dataset just to make the system robust. (I trained YOLO with 200 images, sometimes the trained model confused and detected the wrong object that is not in any of classes, this happens when training with 2000 images per class? This is due to a small number of dataset or it is because of not including those images with no bounding boxes)?

Thank you in advance!


texts: How does region proposal network (RPN) and R-CNN works?

texts: I would like to know where I could find useful articles/papers about the basic concepts of RPN and R-CNN.


texts: Understanding Transfer Learning of Word Embeddings

texts:
I can't quite visualize how transfer learning of pre-trained word embeddings is useful in an NLP task( say named entity recognition ) . I'm studying Andrew NG's Sequence Models course and he seems to say if the training set for the target task is very less, then transfer learning of word embeddings is helpful in a way that unknown words in the training set can be handled in the application .
Let's consider the task of Named Entity recognition ,
My question is , what does the very small training set for the target task contain ? Are they word embeddings or sentences labeled with entities ?
Does he seem to suggest that if the training set is of just  labeled sentences whose words have embeddings in the pre-trained model , then words which are not present in the training set but are closer to those already in the training set also get captured effectively in the application ?
Eg : Consider 'Orange' is in training set . But , 'Apple' is not .
So , in the sentences , ' I like Orange Juice ' and ' I like Apple Juice ' , Apple gets recognized as a fruit , even though it's not in the training set since it is closer to Orange .
Am I right in my assumption ? Or can someone please correct and explain to me if I am not ?


texts: R data frame create a new variable which corresponds to one of the existing one

texts: Currently this is what I get. "community_area_clean" is the new variable that I added by extracting a variable that lists all the communities from a data frame. "CommArea_Name" is the original "unclean" variable. But I find that the names in "community_area_clean" do not correspond to the names in "CommArea_Name". How can I fix it? Thanks.


texts: Dummy encoding the categorical variables using the changed version of OneHotEncoder

texts:
This is my code, I was trying to dummy encode the first column of X using OneHotEncoder but it was showing error and the documentation page of OneHotEncoder says that it has been changed and I wasn't able to figure out how I can do it using the changed version. the data type of X is numpy.ndarray.

from sklearn.preprocessing import LabelEncoder, OneHotEncoder
onehotencoder=OneHotEncoder(categorical_features = [0])
X[:, 0]=onehotencoder.fit_transform(X).toarray()


texts: How to tune bandwidth in machine learning kernel model?

texts:
Gaussian kernel

$k(x,y) = \exp(-\lVert x-y \rVert^2/\sigma^2)$

has a hyperparameter $\sigma$.

I know grid search cross validation, but this would require a lot of computation since computational cost of kernel method scales with the number of samples to the power of 2.


texts: Python and data analysis how can we transform a json array into rows having json properties as columns titles?

texts:
I need to create a recommendation system for a small company using its 500 first orders.

I received a .json file having the following structure:

{
    "data":
    [
        {
            "order_id": 1,
            "seller_id": 1,
            "customer_id": 4,
            "order_status_id": 6,
            "product_id": 2591,
            "quantity": 4
        },
        {
            "order_id": 1,
            "seller_id": 1,
            "customer_id": 4,
            "order_status_id": 6,
            "product_id": 2592,
            "quantity": 1
        },
        {
            "order_id": 2,
            "seller_id": 19,
            "customer_id": 5,
            "order_status_id": 6,
            "product_id": 1025,
            "quantity": 3
        },

        ...

I am new to machine learning and data analysis using python. What I need is group each order_id as a line a dataframe, where it contains:

order id 1    product 1    product 2    product 3

order id 2    product 3    product 4    product 6   product 125

This is for the apriori algorithms.

But for the data analysis, how can split each row into a line with columns in a series or data frame.

I did the following:

import pandas as pd
import numpy as np

data = pd.read_json('orders.json')

And I tried this:

data = pd.read_json('orders.json')['data']

But what I've got is the following:


texts: Clarification about Octave data size limit

texts:
I'm just starting to work on a relatively large dataset after ML course in Coursera.
Trying to work on https://archive.ics.uci.edu/ml/datasets/YearPredictionMSD.
Got an accuracy of 5.2 in training and test set with linear regression using gradient descent in octave.

I tried adding all possible quadratic features (515345 instances and 4275 features), but the code just won't stop executing in my HP Pavilion g6 2320tx, with 4GB RAM in Ubuntu 14.04.

Is this beyond the data size capacity of Octave ?


texts: Is it a red flag that increasing the number of parameters makes the model less able to overfit small amounts of data?

texts: I'm training a deep network (CNN-LSTM-CRF) for Named Entity Recognition. Is there a reason that increasing the number of parameters would make the network less able to overfit a small training set (~20 sentences), or does this indicate a serious bug in the code?


texts:
I'm trying to run H2O in databricks. However, when I do the following:

hc = pysparkling.H2OContext.getOrCreate(spark)

I get the following error:

java.lang.AbstractMethodError

Does anyone know what could be the problem? Thanks!


texts: Recurrent neural network multiple types of input Keras

texts:
For a project I want to use recurrent neural networks, however my knowledge on this subject is still somewhat limited. I do have some experience with convolutional nets and traditional neural networks. I need to predict a probability distribution over one of the inputs of the next step. Most of my sequences are relatively short, with some big outliers in there. There is more than enough data so that should not be an issue.

I'm using Keras for this task, using either LSTM or GRU layers.

My input exists of sequences of categorical input $x_A$, $x_B$ and a few numerical values $x_1$ to $x_{11}$. I'm trrying to predict $x_A$ of the last step that is not trained on. For the high cardinality categorical features I'm using an embedding layer to map them to a dense space and then merge the three different types of input into one layer. My issue is merging them together before going to the softmax layer to do the one-hot encoding prediction of the last step. I cannot seem to get the inputs to match. I pad the shorter sequences to 10 steps plus the label and truncate the longer sequences cutting off the start. I've attempted both the Graph and the Sequential interface.

def build_graph_model(n_A, n_B, max_length, batch_size):
    g = Graph()
    g.add_input(name='A', input_shape=(max_length, ), dtype='int')
    g.add_node(Embedding(n_A, 32, input_length=max_length, mask_zero=True), name='embedding_A', input='A')
    g.add_input(name='B', input_shape=(max_length, ), dtype='int')
    g.add_node(Embedding(n_B, 32, input_length=max_length, mask_zero=True), name='embedding_B', input='B')
    g.add_input(name='rest', input_shape=(max_length, 11))
    g.add_node(Dense(30, activation='relu'), name='rest_dense', input='rest')
    g.add_node(LSTM(100), name='lstm', inputs=['embedding_A', 'embedding_B', 'rest_dense'])
    g.add_node(Dense(n_dest, activation='softmax'), name='softm', input='lstm')
    g.add_output(name='output', input='softm')
    g.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

This gives an assertion error for having the wrong input dimensions. If I remove the $x_1$ to $x_11$ thing completely it works. If I change the Dense layer into TimeDistributed(Dense) I still get the same error although I think that should work. Inserting the rest input directly into the merge doesn't work because the merge cannot be the first layer. I'm uncertain how to fix this issue.

EDIT: This guy asks the same question but much more concise (https://groups.google.com/forum/#!topic/keras-users/zCuS4xdv5QY)


texts:
Here is the problem
The initial four cluster partition {c1, c2, c3, c4} for the text collection is provided by this link text vs cluster. Assuming that
the ground-truth partition is given by

cacm texts belong to cluster1
cisi texts belong to cluster2
cran texts belong to cluster3
med texts belong to cluster4

build the confusion matrix (CM) for the partition provided. That is build a 4X4 matrix whose row i shows distribution of ci elements
among cluster1, cluster2, cluster3, and cluster4. For example the first row is computed as follows:
CM11=|c1∩cluster1|,  CM12=|c1∩cluster2|,  CM13=|c1∩cluster3|, and CM14=|c1∩cluster4|
I understand the problem just not sure how to code it up.
I was thinking along the lines of this

//open text file
fileID = fopen('list.txt');
C = textscan(fileID,'%s %s');
fclose(fileID);

What this does is it orders the names as $C\{1\}\{j\}$is the file name and $C\{2\}\{j\}$ is the cluster name.
By the way I am using Matlab


texts:
I have several computing devices. I have used an algorithm to balance the load between these devices. There is a central coordinator which controls the load on each device and if one device is overloaded, it will transfer some load of overloaded device to a less-loaded device. I have considered the number of tasks on each device as the load of that device.

I have measured the response time with load balancing and without load balancing and now I want to show the difference between these situations on a graph. I know that y-axis represents "response time", but the problem is that I can't figure out what I should consider for x-axis. Each device may have several tasks, for example device1 = 3, device2 = 1 and device3 = 10 tasks. Should I use the mean of these values for x-axis value? If not, what should I can use for x-axis?


texts: What does an Input layer of shape=(None,) or (None,12) actually mean?

texts: Is this telling the model that there are two dimensions (i.e. it’s a matrix) but we don’t yet know the size of that particular dimension? If so, how can the model be compiled? Doesn’t the size of each dimension affect the number of nodes in middle layers?


texts: Similarity between an n-dimensional curve and selected subsets of them

texts:
I'm asking here to get some advice. My goal is to detect similar patterns in an n-dimensional data. For example (extreme simple) you have two axes: X-Axis is Time (in seconds) and Y-Axis is Power (in Watt). So you get time-series with a specific pattern. My goal is to select a subset of the time-series and calculate the similarity of other input data to the selected subset.
In the end, I want to define multiple patterns and the output predicts which pattern comes most closely to the given input data.

Is it suitable to use Machine Learning e.q. pattern recognition for that problem or is it a pure overkill? (I got the advice to use ML; also under the condition that I have an generic n-dimensional data)
If ML is suitable for this problem, can you give me some advice where I could start? I don't ask for a solution in any way, I just want to have some suggestions in which direction I should look, because ML is a very new topic for me and I don't have much experience in practice.

Thank you.


texts: Batch-mode active learning: How to select the batch that will bring the largest model improvement

texts:
I am facing a problem where I want to use active learning to improve my classifier. Basically, I can choose data from one (and only one) data set among a set of candidate data sets. The question is which one to choose?

In other words, given a set of candidate data sets that I can use improve my classification model, which one is going to improve the model most?
Can I use the some metric (eg., average) from the objective functions of the batches? Do I need to normalise the objectives across all data sets? Have metrics to infer the magnitude of model improvement been proposed?

At this stage, my objective function looks like:

a  Uncertainty + (1-a) Diversity

where a is a weight factor, Uncertainty is the uncertainty in the model prediction for a given data point and Diversity is a measure of distance between the data point and the training population.

Any help would be greatly appreciated.

Cheers

Franz


texts: Can preprocessing the whole population cause data leakage?

texts:
Introduction

I understand the problem of data leakage that could be caused by the preprocessing step when our training and test sets are just samples of an unknown population. The preprocessing parameters should be calculated from the training set only, then we just apply the same procedure to validation/test set, since this would be the way to proceed with any other sample from the unknown population (in production stage, for example).

Question

What about the situation where we have the whole population at hand? Could we calculate the preprocessing parameters (scaling factors, encoding, etc.) from the entire population?

Extra Context

We have the whole population and the modeling process would depend of user input. The training set is defined by the user input and the trained model is used to classify the population.


texts: is it necessary for Artificial NN to be fully connected or only fully connected NN is called ANN?

texts: is it necessary for Artificial NN to be fully connected or only fully connected NN is called ANN ?


texts: How to read file from user in Shiny and assign it to a variable in global.r?

texts:
I want to read a csv file as input from user in Shiny and assign it to a variable in global.r file.The code I have in ui.R is

fileInput('file1', 'Choose CSV File',
                     accept=c('text/csv', 'text/comma-separated-values,text/plain', .csv'))

The code I have in main Panel of server.R is

textOutput('contents')

The code I have currently in server.R is

output$contents <- renderText({
      if(is.null(input$file1))return()
      inFile <- input$file1
      data<-read.csv(inFile$datapath)
  print(summary(data))

 })

I want to assign this input$file1 to a variable called 'data' in global.r. Please let me know if this is possible.
Thanks


texts: John10 , michaelscott, James.white , Jr.Jones , James-Anderson ,WhiteWalter10 -- These are some of the different cases of usernames possible(there may be more ). I have about 200K such usernames . I need to extract the first name from all of them. If only first name not possible then atleast only the relevant names (for example - WhiteWalter10 should give only Walter or White and Walter ). I already have a dataset of surnames so I can filter the first name


texts: Diff. in P-value & F-Stat. Multiple linear regression

texts:
Even if we have individual p-values for each predictor.

Why do we need overall F-statistic?

I read this solution but I am not sure if I get it right. Can someone please explain?

Source: "An Introduction to Statistical Learning: with Applications in R" by James, Witten, Hastie and Tibshirani


texts: Visualizing 28 different variables with 28 different colors?

texts:
ColorBrewer seems to be very useful in selecting a color pallet to represent factors that have up to 12 possible values.

I have 28.

Is it a horrible idea to represent 28 variables with color? If so, could you suggest an alternative visual indicator?

Currently I'm using the colors for column side colors in a heatmap shown below. As you can see, the Strain column is not very informative:


texts: Test independence based on Kernel Density Estimation

texts:
I am working on a problem where I have a dataset of
$X$ is dataset with $(X, Y, T, K)$ four attributes, I'd like to test if $P(X, Y, T)P(K) = P(X, Y, T, K)$, that is if $X, Y, T$ is independent of $K$. I have two questions:

Is it possible to use kernel estimation to fit $P(X, Y, T)$ $P(X, Y, T, K)$ and $P(K)$ respectively and test the independence? If I did so, will the output be accurate? (My dataset is very large with tens of millions of records. So it probably won't underfit.)
If not, what I can do to test it?

To convert to a Hypothesis Testing setting:
$$H0: P(X, Y, T)P(K) = P(X, Y, T, K)$$
$$H1: P(X, Y, T)P(K) \neq P(X, Y, T, K)$$
I don't know how to test this.

Any help or insight appreciated!!


texts: Pearson correlation on two categorical variables

texts:
I am using the fourth-corner method in one of my papers (for those who need the name). The method was developed to test associations between variables in two datasets. In my case, the datasets contains traits of species (e.g. trait Size with modalities 'small', 'medium', 'large'). The method recognizes the data type and then apply appropriate statistics.
The correct cases:

If two variables are quantitative, the fourthcorner calculates Pearson correlations.
If two variables are qualitative, factorial, the method calculates a Chi2.
If the variables are mix, it calculates a Pseudo-F test.

However, in my study and a study I criticize, we had to convert factorial data into categorical binary data.
In my case, instead of having ONE trait column containing small, medium, large, I have 3 columns, small, medium, large coded as factors yes or no.
In that other study, they coded as 0/1. If coded as 0/1, the data is first standardized.
If categorical and coded as  0/1 instead of factorial (e.g. biological traits of species, coded as presence/absence 0/1), the method will calculate Pearson correlations instead of Chi2. Or actually, "will return Pearson correlations" if wrongly specified in the output. Which is what they did and analyze in the study.
I am trying to justify why this is wrong, why you should use categorical data coded as yes-no and do a Chi2, but I am not entirely sure how to explain it, or justify how correlations calculated from the 0/1 do not mean what one thinks.
If we have two categorical variables, height and weight, split in class such as:
height: short, medium, tall (0/1)
weight: thin, large (0/1)
To my understanding:
When you calculate correlation between short and thin, you calculate correlations of being short/not short, with thin/not thin as the variables are non-dichotomous.
How would you justify that?
Tables look like this:
for their data

Short
Medium
Tall

0
1
0

1
0
0

0
0
1

Thin
Large

0
1

0
1

1
0

for my data

Short
Medium
Tall

no
yes
no

yes
no
no

no
no
yes

Thin
Large

no
yes

no
yes

yes
no

Best,


texts: Will reinforcement learning work if states wont get repeated again?

texts: I am working on a information retrieval model where the user enters a query and the model has to retrieve 3 most relevant FAQ pairs.I am collecting implicit feedback in terms of page clicks etc.What I don't understand is how to define states if I want to implement this with reinforcement learning?If I define each query as a state, all states will be different .Can anyone shed some light on this please!!!


texts: Defining objects in training data for deep learning

texts:
I am performing end-to-end deep learning in ArcGIS Pro 2.5, following a similar procedure to the one set out in this tutorial: https://learn.arcgis.com/en/projects/use-deep-learning-to-assess-palm-tree-health/

I am trying to train the model to recognise bats in thermal orthomosaics taken from a drone. The bats are much warmer than their background, so are detectable in the images as bright circular objects with a diameter of approximately 5 pixels, against a dark background (see image 1).

So far, I have been defining training samples by drawing a circle from the center of each bat to just outside the edge of the animal.

However, this is not giving me great results.

Is anyone able to offer advice on the best way to define a bat in my training images? Should I exclude all of the dark area surrounding the bat? Should i use the square training sample drawing tool instead?


texts:
Given two sentences, I want to quantify the degree of similarity between the two text-based on Semantic similarity.
Semantic Textual Similarity (STS) assesses the degree to which two sentences are semantically equivalent to each other.
 say my input is of order:

index    line1                       line2
0        the cat ate the mouse      the mouse was eaten by the cat
1        the dog chased the cat     the alligator is fat
2        the king ate the cake      the cake was ingested by the king

after application of the algorithm , the output needs to be

index    line1                       line2                           lbl
0        the cat ate the mouse      the mouse was eaten by the cat    1
1        the dog chased the cat     the alligator is fat              0
2        the king ate the cake      the cake was ingested by the king 1

Here lbl= 1 means the sentences are semantically similar and lbl=0 means it isn't.
How would i implement this in python ?
I read the documentation of bert-as-a-service but since i am an absolute noob in this regard I couldn't understand it properly.


texts: Implementing a CNN with one convolution layer

texts:
I am trying to implement a 1 channel CNN by slightly changing this article: this article.
The problem is that I am new to keras and deep learning and I don't know this far why I am getting this error:

ValueError: Negative dimension size caused by subtracting 100 from 1 for 'conv2d_1/convolution' (op: 'Conv2D') with input shapes: [?,1,70,100], [100,100,100,64]

Obviously, it's a mismatch in the dimensions.

I am using this code code:

from keras.layers import Embedding
from keras.layers import Conv2D
from keras.models import Sequential
from keras.layers import MaxPooling2D
from keras.layers import Reshape
import pdb
Vocab_Size=11123
MAX_SEQUENCE_LENGTH=70
EMBED_DIM=100
model = Sequential()
embed1=Embedding(Vocab_Size+1,EMBED_DIM,input_length=MAX_SEQUENCE_LENGTH,input_shape=(MAX_SEQUENCE_LENGTH,EMBED_DIM,1))
nb_labels=6
model = Sequential()
model.add(embed1)
model.add(Reshape((1,MAX_SEQUENCE_LENGTH, EMBED_DIM)))
model.add(Conv2D(64, strides=5, kernel_size=EMBED_DIM, activation="relu", padding='valid'))
model.add(MaxPooling2D((MAX_SEQUENCE_LENGTH-5+1,1)))
model.add(Flatten())
model.add(Dense(256, activation="relu"))
model.add(Dropout(0.3))
model.add(Dense(len(nb_labels), activation="softmax"))

model.compile(loss='categorical_crossentropy',
                optimizer='rmsprop',
                metrics=['acc'])

Edit1:
I updated as mentionned by Media padding to same.
Now I have another problem for the next layer:

ValueError: Negative dimension size caused by subtracting 66 from 1 for 'max_pooling2d_1/MaxPool' (op: 'MaxPool') with input shapes: [?,1,14,64].


texts: Parameter Adjustment based only on tagged predictions

texts:
not sure that this is the best place to post this but if not, please let me know if there is a better stack community.

I have an anomaly detection method which has some parameters. I have some data that I need to find anomalies for. This data is not tagged. I use the anomaly detection method on the data using some default parameters. It predicts some anomalies. The user looks ONLY AT PREDICTED ANOMALIES and determines if he or she agrees with them or not (is the predicted anomaly truly an anomaly?) Now I need to adjust the parameters of the anomaly detection method so it can do better next time whether that is on the same data set or a new data set.

More specifically:

1) system predicts some anomalies

2) user looks at those predicted anomalies and says if they agree or disagree with them

3) system looks at those predicted anomalies and whether or not the user agrees with them and makes adjustments to the parameters based on this information.

The ``making adjustments" part is what I am looking for ideas about. As for why I am not listing a specific method, I was hoping for more generic ideas.

How do I go about this (the parameter adjustment) for either situation (same or new data set)?


texts:
Boosting is a sequential technique in which, the first algorithm is trained on the entire dataset and the subsequent algorithms are built by fitting the residuals of the first algorithm, thus giving higher weight to those observations that were poorly predicted by the previous model. Examples are adaboost and GBM, etc. My question is that how to perform boosting when ensembling the base learner? Especially how to get the residual if it is a classification problem?

I know how to bagging the base learners and stacking the base learners. I just have no idea how to boosting the base learners.

Thanks!


texts: What are available Python libraries for Interpretable ML?

texts: I recently become familiar with Interpretable ML and I found some libraries like LIME. I would be thankful if you can suggest to me some libraries and what are the advantages of each library.


texts:
I am a student and I am studying machine learning. I am focusing on neural network, and I have seen that for a neural network, to define the optimal weights, we don't use gradient descent, but we use stochastic gradient descent.
As far, I have unserstood that the main difference is that for gradient descent we perform the iterative step of computing the gradient and adding it to the current weight for the whole dataset, while for stochastic gradient descent we compute the derivative and update the weights for only one datapoint at a time (or for batches of datapoints), so we have something that looks like this:

teying to go deeper in the topic, I have found that we use stochastic gradient descent because the loss function is not convex, but I don't understand why it should not be convex, and also I have found another explaination which says that we use sthocastic gradient descent because it reduces the complexity in computations.
All these explainations gave me a lot of doubts, and I am a little bit lost.
So, Why do we use stochastic gradient descent in neural networks and what are the main ideas behind this optimization technique?


texts: Using already trained classifiers in Orange

texts:
I know that with Orange it is possible, using the 'Test & Score" widget, to test the performance of a classifier trained on a specific training set using a different test set ('Test on test data' option).

My question is: is it possible to use a training set to train a classifier, tune its hyperparameters through cross validation on a separate validation set, and, finally, test the classification performance on a third, separate set using the already trained classifier?

How this can be achieved?


texts: ROC and AUC curve for CNN multi-class classification problem

texts:
I have produced a convolutional neural network to classify images (malware images) into different classes/families. I have managed to produce a confusion matrix and classification report. My supervisor for my project wants me to output a ROC and AUC graph, but ROC and AUC graphs are only really for binary classification problems when reading online articles; this is what it tells me.
I am new to this area but thought I'd spend my weekend giving this a go. So far, nothing online has been helpful.
My model predicts on a dataset with 9,339 images; however, the dataset is unbalanced, but I have addressed this in the model. In total, there are 25 different classes/Families an image could belong to. I have read up that you can produce a ROC graph for every possible class.
path_to_data = "dataset"

train_test_generate_batches = ImageDataGenerator()
batch = train_test_generate_batches.flow_from_directory(directory=path_to_data, target_size=(64,64), batch_size=10000)
imgs, labels = next(batch)
X_train, X_test, y_train, y_test = train_test_split(imgs/255.,labels, train_size=0.7, test_size=0.3)

num_classes = 25

model = Sequential()

#input layer
model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=(64,64,3)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(16, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.25))
model.add(Dropout(0.5))
model.add(Dense(50, activation='relu'))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer = 'adam', metrics=['accuracy'])

y_train_new = np.argmax(y_train, axis=1)
class_weights = class_weight.compute_class_weight('balanced', np.unique(y_train_new), y_train_new)

history = model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10,  class_weight=class_weights)
scores = model.evaluate(X_test, y_test)

Could anyone explain how I could produce the curves from this neural network? Any help would be appreciated.


texts: Encoding features like month and hour as categorial or numeric?

texts:
Is it better to encode features like month and hour as factor or numeric in a machine learning model?

On the one hand, I feel numeric encoding might be reasonable, because time is a forward progressing process (the fifth month is followed by the sixth month), but on the other hand I think categorial encoding might be more reasonable because of the cyclic nature of years and days ( the 12th month is followed by the first one).

Is there a general solution or convention for this?


texts: How to build a hybrid graph with CNN and RNN Architecture?

texts: For an instance, we have M matrix of size n*k, I would like to treat each matrix independently and perform the convolution operation at each one in order to introduce to an LSTM layer the outputs of the convolution layer. can I treat each matrix independently or do I have to use the channel technique and how I can provide each LSTM cell? you find the architecture link below, thank you in advance!!


texts:
I am trying to implement PReLU activation in TensorFlow 2.4.1 as given here.
Got the following error:

ValueError: Variable alpha already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope?

Here is my code:
def prelu(_x):
  alphas = tf.compat.v1.get_variable('alpha', _x.get_shape()[-1], initializer=tf.constant_initializer(0.0), dtype=tf.float32)
  pos = tf.nn.relu(_x)
  neg = alphas * (_x - abs(_x)) * 0.5

  return pos + neg

Any help is appreciated.
Note :
I do not want to use layer interface tf.keras.layers.PReLU, because it cannot be passed as an argument to conv2D, like below
Conv2D(filters, (3, 3), padding='same', activation='prelu')


texts:
I have been given this code below to modify to optimize the value of theta over vector (0.01,0.02, ..., 0.99, 1.00) where R0=10 and D=14, any help at all would mean a lot. Thanks
errorSS<-function(epidemiology.data,N=1000000,R0,D,theta,N.time.steps=2100,delta.t=0.1){

gamma<-1/D

beta<-R0*gamma/N

S0<-theta*N

I0<-epidemiology.data$No.infected[1]

sample.index<-epidemiology.data$Time.index

S<-numeric(N.time.steps+1)

I<-numeric(N.time.steps+1)

S[1]<-S0 # Initial conditions

I[1]<-I0 # Initial conditions

for (i in 1:N.time.steps){

S[i+1]<-S[i]-beta*S[i]*I[i]*delta.t

I[i+1]<-I[i]+beta*S[i]*I[i]*delta.t-gamma*I[i]*delta.t

}

errorSS<-sum((epidemiology.data$No.infected-I[sample.index])^2)

return(errorSS)
}


texts: Can the output of convolution on image be higher than 255?

texts:
I have probably a very simple question. When I convolve an (grayscale) image using some kernel I get some output. The original pixel can be only between 0 and 255. Is possible that the output of a convolution can be higher? Because we are creating feature map, which I understand is another image. Is the output only up to 255, do we scale it down or it does not matter?

Thanks.


texts: semi-structured text parsing using machine learning

texts:
I am looking for a method to parse semi-structured textual data, i.e. data poorly formatted but usually having a visual structure of a matrix which may vary a lot in content and number of items in it, which may have headers or not, which may be interpreted sometimes column-wise or row-wise, and so on.

I have read about the WHISK information extraction paper : https://homes.cs.washington.edu/~soderlan/soderland_ml99.pdf

but unfortunately, it is not very detailed and I have not been able to find a real-system implementing it, or even snippets of code.

Has anybody have an idea where I can find such help? Or suggest an alternative approach which may be suited to my problem?

Thank you in advance for your reply!


texts: Reciprocal rescaling of product of two matrices

texts: I read in many papers about product of two matrices being invariant to reciprocal rescalings.  What exactly does this means ?


texts: Which algorithms should I use for recommendation system using a graph database?

texts:
Basically I'm developing a recommendation system using a graph database (specifically neo4j), and I want to apply recommendation algorithms. Since i'm using a graph database, I can see the recommendation problem as a graph problem, and intuitively i can use graph based algorithms for the recommendation system.

From my research, recommendation systems are a subclass of information filtering system that seek to predict the "rating" or "preference" that a user would give to an item. And there exists basically two types, collaborative filtering and content based.

I've done a research on the algoritms, and i found some interesting ones:

Weighted Bipartite Graph algorithm
Energy Spread Activation
Union Colors

My question is simple, which other graph algorithms exists that can be used for graph based recommendation system? Or if I use a graph database for recommendation system, the algorithm doesn't necessary need to be a graph based?

Thanks. Any suggestions are welcomed.


texts:
I am a little new at this - I am used to just querying data and not so much analysis of the data so any help would be greatly appreciated.

I have some data that is trending month over month. Some months with an increase in volume and others with a decrease. The goal is to identify a "spike" in the volume. Basically, is the increase abnormally large. I was trying to figure out the best way to do this. I thought the best thing to do was take the standard deviation and anything outside of two deviations - BAM!, that's a spike. Then I thought about it some and I got concerned about what happens next month. That spike is still in that data, it happened, but doesn't it shift the standard deviation so that the next month may not show a "spike" when without the prior "spike" it could have? For continuous data maybe this is less an issue but my data is rolling, I only have the last 6 months of data. If there is a "spike" this month and every one is put on notice, and the level never changes, then in six months wouldn't it seem like there is no spike?

I guess my question is, how would you quantify a measure for a "spike" given this limitation?


texts: How does class_weights work in RandomForestClassifier

texts:
I'm facing a problem with unbalanced classes, and have tried out a couple of methods like over and under sampling. However, my cross validation mean comes out to be only 0.4 and my confusion matrix shows that the prediction and recall is completely incorrect.

Ive read that the next step is to add weights to my class; I have two of them "Won" and "Lost". In Pandas, how am I supposed to assign weights to them? I know that there is a "class_weights" attribute, but I have no clue on how to use it.

Thanks

PS. My "Won" class is unbalanced, very small compared to the "Lost" one. I train by repeating the set of "Won"s twice and randomly sample an almost equal amount of "Lost"s. I've tried all sorts of combinations of the classes.


texts: Not sure how to visual customer retention vs product line count

texts:
I created a customer retention grid that tracks customer counts by territory by year. The sales managers have loved it. The numbers were terrible, but now they know what they didn't know and can address it.
The follow up question that the report generated was calculating customer retention vs product line count. We have five major product lines and the sales managers want to see if there's a correlation between the number of product line purchases and retention, which I think is a great question. The assumption is that if they only buy one from one product line, the retention is low, but if we get the whole system in place and they buy 4 or 5, the retention is high.
What I'm stuck on is how to group and display the data.

Customer A buys 1 product line in year 1, then 3 in year 2, and then 2 in year 3, and then 0 in year 4.
Customer B buys 1 product line in year 1, then 1 in year 2, and then 0 in year 3, and then 0 in year 4.
Customer C buys 2 product lines in year 1, then 2 in year 2, and then 2 in year 3, and then 1 in year 4.
Customer D buys 2 product lines in year 1, zero in the rest

My initial thought is to average the counts and then graph based on that.
Is there a better / fancier / cooler way to calculate what I'm trying to achieve?


texts: Large Graphs: NetworkX distributed alternative

texts:
I have built some implementations using NetworkX(graph Python module) native algorithms in which I output some attributes which I use them for classification purposes.

I want to scale it to a distributed environment. I have seen many approaches like neo4j, Graphx, GraphLab. However, I am quite new to this, thus I want to ask, which of them would be easy to locally apply graph algorithms (ex. node centrality measures), preferably using Python. To be more specific, which available option is closer related to NetworkX (easy installation, premade functions/algorithms, ML wise)?


texts: Can you use precision/recall/f1 scores in a balanced data context?

texts: I want to evaluate two classifiers. The data is perfectly balanced and there is the same amount of data per class. Since it is balanced I am using accuracy. I am interested in assessing which classifier yields more correct predictions. Would it make sense to include the other basic metrics - i.e. precision, recall and f1-score? What are the best measures in this scenario?


texts: How to place XGBoost in a full stack for ML?

texts:
Is XGBoost complete by itself for prod-strength machine learning?  If not, with which other tools or libs is it typically combined, and how?

(I recently read a description of a stack that included ca 5 pieces, including XGBoost and Keras.)


texts: Spark 1.6.1 - Determining the number of clusters in a data set

texts:
There existis any way to find the number of cluster needed to use in K-Means?

I'm having some difficulties to find a method to determine k...

Many thanks!


texts: Constructing a Maximum Entropy Classifier for Sentence Extraction

texts:
So I'm reading this paper which uses a max ent classifier for sentence extraction. The parametric form for a conditional max ent model is :
$$ P(c|s) = \frac{1}{Z(s)} \exp \sum_i \lambda_if_i(c,s)$$
where $f_i(c,s)$ is a feature which has a weight $\lambda_i$ associated with it.
Now, the paper states that conjugate descent was used to find the optimal set of weights (page 3) - and this is what I'm unable to comprehend. How to use/apply conjugate gradient descent to calculate the optimal set of weights?


texts: What is max_trials and executions_per_trial in keras-tuner

texts:
I am wondering what is the difference between max_trials & executions_per_trial in kerastuner.tuners.bayesian.BayesianOptimization function. Does execution_per_trial somehow relates to cross valdation? max_trails at first sounds like number of epochs to train but its not the case since there we have a another attribute for that.  I didn't get much from the original documentation found here

  You should specify the model-building function, the name of the
  objective to optimize (whether to minimize or maximize is
  automatically inferred for built-in metrics), the total number of
  trials (max_trials) to test, and the number of models that should be
  built and fit for each trial (executions_per_trial).


texts: How does convolution operation perform in CNN?

texts:
I know that convolution operation has the property, associative law. if I need to use multiple filters in succession and to perform this operation on multiple images, it makes sense to convolve the multiple filters into a single filter ahead of time for a computational benefit. But, I wonder if the convolution layer in CNN is really computed by convolving the multiple filters into a single filter ahead.

To be honest, I just know the reason why the convolution filter is used instead of the correlation filter.


texts:
Premise:
I often get files from colleagues that I need to work on. Often times, these files have spaces in the names. Working with these files at the command line or in scripts can be tedious.

Possible solutions:
With the rename program (on nix systems), I can easily rename these files, eg:

$ rename --sanitize --lower-case *

I recently found that rename can actually just create a link to the original file, leaving the original filename unchanged:

$ rename --sanitize --lower-case --symlink *

Question:
What are the (potential) pros/cons of these two methods?

For example, it seems that creating a bunch of symlinks has the drawback of making my filesystem 'messier', but renaming the files has the draw back of trying to match up the files that I'm using with my colleagues (whether I'm redistributing my code or just communicating "I did [analysis] on [file_x.csv]").

Additional info:

Generally I'm the only person actively working on these files, but it is important that my work be archived so that other people can refer back to it or re-analyze the data in any way of their choosing. I work in an academic setting, so, in principle, the raw data and my methods should be archived indefinitely.


texts: Input changing row count matrices in an MLP

texts:
I want to input a numpy 2d array into MLP but I have an array of 50395 rows that contains many 2d array of shape (x, 129). x because some matrices have different row numbers. Here is an example :
train['spec'].shape
>>(50395,)
train['spec'][0].shape
>>(41, 129)
train['spec'][5].shape
>>(71, 129)

Here an snippet of my code :
X_train = train['spec'].values;     X_valid = valid['spec'].values
y_train = train['label'].values;    y_valid = valid['label'].values
model.add(Dense(12, input_shape=(50395, ), activation='relu'));
model.fit(X_train, y_train, validation_data=(X_valid, y_valid), epochs=500, batch_size=1);

I get this error on last line (model.fit) :
ValueError: Error when checking input: expected dense_54_input to have shape (50395,) but got array with shape (1,)
How to fix this problem so that the network can take as input all 50395 matrices of shape (x, 129)?


texts: Why is having high p-value makes dependent variable significant in backward elimination?

texts:
In backward elimination, I heard the steps of fitting the model by keep removing the highest p-value(a.k.a. insignificant independent variable) each time like below

  Select a significance level to stay in the model(e.g. SL = 0.05)
  Fit the full model with all possible predictors
  Consider the predictor with the highest P-Value(P > SL)
  Remove the predictor
  Fit model without this variable (Repeat step 3-5 until P <= SL)

But the part which I don't get is why is having higher p-value makes the corresponding independent variable insignificant. Doesn't having high p-value mean it's more close to the null hypothesis so that that variable is more significant?


texts: Python: forecast unevenly spaced time-series?

texts:
My data has timestamps corresponding to the failure occurrences of a specific component in machinery. The timestamps are not uniformly distributed.
My question is:
1) what methods can I use to (almost) accurately to forecast future occurrences (timestamps) of Failure?
2)What other features can I derive?

What I've tried so far:

Since the timestamp sequence is unevenly spaced I've derived a feature datediff= difference between sequential fault occurrences.
Since it is now a univariate time-series I have tried classical time-series forecasting methods like ARIMA and SARIMA (hasn't worked out well)

I am posting the seasonal decompositions of the time-series
freq=7(weekly)

freq=30

acf/pacf


texts: How to handle continuous values and a binary target?

texts:
This is going to be a very beginner's question.

I have a datset of continues features like LoanAmount, LoanDuration(multiclass?), ... ClientIncome, ClientFreeSources, etc. and a binary target whether a contract was sued or not.

I'm not sure how to approach the problem as I'm fairly new to DS.
I can reformulate the target into limited number of values like 5, 11, 18, ... which I think would be probably precieved as multiclass by a model or into continues value expressed in DebtAmount after some work at the SQL Server end.

However, the simple flag Sued 1/0 would be prefered option at the begining if possible.
I also wonder how to treat high-cardninality categorical features like post code because the number of new dummy variables which seems to me too high.

Thanks for your answers in advance.


texts: How can we create neural net to detect false predictions?

texts:
I created a convolutional network to recognise certain substrings.

For example, the following phrases would be mapped to the "What" class:

What are you doing?
What you doing?
Whatcha doing?

The following to the "How":

How are you doing?
How you doing?

Now, say I have millions of examples and thousands of classes, my network learns relatively well. However, sometimes a new "phrase" that belongs to a never seen class appears. Of course, the model cannot map it to something it has never seen, however, when I look at the output layer values, I was hoping to see all neurons to have small values. My idea was that, I would use the softmax and then if the max value is less than 0.95 or some high value, I would assume the network is unsure, therefore remove that result for manual observation.

However, in many cases, I have seen my network to be almost sure (a neuron with value close to 1) even when it has never seen that class.

For example, say I have the two classes What and How, I would expect:

Whatcha doing? --> "What"= 0.95, "How"= 0.05
How you doing? --> "What" = 0.02, "How"= 0.98
Where you going?--> "What"= 0.6, "How"=0.4

In the last case, the network is not too sure on either, and I would check the input. Unfortunately, this is not working, and I am getting high values (close to 1) neurons for some new input.

How can I fix this? Please, bear in mind, I am happy to sacrifice some accuracy in order to ensure the network never tells me it is sure about new inputs, in other words, I am happy to go and manually check more inputs (even if seen before) rather that let even one unseen input go unnoticed.


texts: Is Max Pooling and Conv used on anything else but images?

texts:
Can you think of any domain of application, other than 2D images, where it could make sense to use max pooling or convolution?

Because the ONNX format allows for non 2D inputs. On the operators page (https://github.com/onnx/onnx/blob/master/docs/Operators.md#MaxPool) they say

  dimensions for image case are (N x C x H x W), where N is the batch
  size, C is the number of channels, and H and W are the height and the
  width of the data. For non image case, the dimensions are in the form
  of (N x C x D1 x D2 ... Dn)

And I did a search, and couldn't find an application where non images.


texts: What is GridSearchCV doing after it finishes evaluating the performance of parameter combinations that takes so long?

texts:
I'm running GridSearchCV to tune some parameters. For example:

params = {
    'max_depth':[18,21]
}

gscv = GridSearchCV(
    xgbc,
    params,
    scoring='roc_auc',
    verbose=50,
    cv=StratifiedKFold(n_splits=2, shuffle=True,random_state=42)
)

gscv.fit(df.drop('LAPSED', axis=1), df.LAPSED)
print('best score: ', gscv.best_score_, 'best params: ', gscv.best_params_)

All fine. Because I've specified some verbosity, it outputs some stuff about what it's doing, like this:

Fitting 2 folds for each of 2 candidates, totalling 4 fits
[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.
[CV] max_depth=18 ....................................................
[CV] ........... max_depth=18, score=0.9453140690301272, total= 8.2min
[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:  8.3min remaining:    0.0s
[CV] max_depth=18 ....................................................
[CV] ........... max_depth=18, score=0.9444119097669363, total= 7.9min
[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed: 16.3min remaining:    0.0s
[CV] max_depth=21 ....................................................
[CV] ........... max_depth=21, score=0.9454705777130412, total= 8.4min
[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed: 24.8min remaining:    0.0s
[CV] max_depth=21 ....................................................
[CV] ........... max_depth=21, score=0.9443863821843195, total= 8.3min
[Parallel(n_jobs=1)]: Done   4 out of   4 | elapsed: 33.2min remaining:    0.0s
[Parallel(n_jobs=1)]: Done   4 out of   4 | elapsed: 33.2min finished

However once it's finished running all the folds, it takes a very long time (at least as long as it takes to fit and evaluate one fold for one combination of parameters) for it to return the output to print('best score: ', gscv.best_score_, 'best params: ', gscv.best_params_), even though I could literally manually calculate that easily from the data it outputs as part of that fitting process. I presume this means that the algorithm is hung up doing something else after it finishes fitting and evaluating the different models, but I'm not sure what that might be.

n.b. actually, the thought just occurred to me that this might be time spent retraining the model on the parameters it identified as giving the best performance, so that it's available to the .predict() etc methods. I'm just checking that now by passing refit=False to prevent that from happening, and if it works I'll answer my own question.


texts:
is it possible to use the train result as another feature and retrain?

for example I make prediction with classification and
add this result to the table and train xgboost?


texts: which forecasting models could be chosen?

texts:
I'm new for data analysis. I got some data from the regional environmental center.

Measurements:
Datetime, PointID, SubstanceID, Value (substances concentrations in air), MeteoID ,NextValue

Meteorological data:
MeteoID, Datetime, temperature, wind speed, wind diretion,humidity, pressure, precipitation.

Substances(8 substances(CO,Cl2,NO2, etc.), 5 of them have about 1.2 million records):
SubstanceID, Name, MaxVal (maximum allowable concentration value).

Points(9 static monitoring stations):
PointID, Adress, Longitude, Latitude.

Measurments table contains about 8 million records of substances concentratios in air(local mysql). I linked the measurement data and the meterologicala data (closest in time), and added the next value to measurement record. time between measurements 20 min( in average, for some periods data are missed).

I want to make a predictive data analysis and get short-term forecasts for substances concentrations depending on the previous value and weather data(may be for few hours). I am still thinking which methods, techniques can be chosen. I want to consider different methods. which toolkit is most suitable for this case? Should the data be considered as a multidimensional time series? I'm currently looking into the direction of some kind of neural network and implementation in python (but I still don’t know which package).


texts: Extract or subset hundreds of columns from a data frame

texts:
I need to extract many columns from a dataset.
I have a very large csv file with thousands of columns and rows, and I read it into R using:

mydata <- read.csv(file = "file.csv",header = TRUE,sep = ",",row.names = 1)

Each column is a gene name. I know how to extract specific columns from my R data.frame by using the basic code like this:

dataset[ , "GeneName1", "GeneName2"]

But my question is, how do I pull hundreds of gene names? Too many to type in? They are listed in a txt file.
I'm new, so please go easy on jargon and abbreviations.


texts: K-Means anomaly detection not clustering anomalies

texts:
K-means anomaly detection scatter plot

The following code, takes a single column from a dataset and then adds 50 anomalies to the dataset that is quite bigger than the maximum values of the dataset.

import pandas as pd
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
import seaborn as sns

X=pd.read_csv('C:/Files/dataset.csv', sep=';', encoding='latin1' )

#Adding the anomalies

for i in range(0, 50):
    X.append(X.my_column.max() * (10 + pd.np.abs(pd.np.random.normal())))
X = pd.np.array(X)

clf = KMeans(n_clusters=2, init='k-means++', max_iter=300, n_init=10, random_state=1)
clf.fit(X.my_column.values.reshape(-1, 1))
X_prd = clf.predict(X.my_column.values.reshape(-1, 1))

plt.scatter(X.index, X.my_column, c=X_prd)

The picture bellow shows the results and I was expecting outlier cluster to be clear compared the normal data.

Why so ?

Because for creating the anomalies I took the maximum value of my_column which was  9689.

I am stuck here and I don’t know where to do from here, so I would appreciate some help.

The goal is that K mean to detected these added anomalies.


texts: Is it possible to deploy a python trained machine learning model (e.g. a .pkl file) in C language?

texts:
I would like to train my machine learning using Python and libraries such as tensor flow, keras, and scikit-learn. After trained, I would like to export this trained model to a file, so far I have been using the library pickle. I feel that this is pretty standard in any ML project.

However, and the point of this question, is it possible to use this trained model (e.g. a .pkl or .sav file) in C language? When I say "use", I mean passing values to the model and getting a prediction back.

This is somewhat similar to TensorFlow Lite for Microcontrollers, however, I am not sure if this is the most appropriate and easier approach to this problem.


texts: Custom learner for sklearn AdaboostClassifier

texts:
I'd like to use scikit-learn AdaboostClassifier differently from the original proposal:

1) I want to mix different types of weak-learners in one Adaboost classifier;

2) I want to use custom learner types, different from the default ones available in sklearn.

After some research, i've noticed that the custom learners must implement a weight sampling function but got nowhere from here. I'd like to know if it's a good idea to customize sklearn or if i'd rather create something from scratch (which will take a lot of time). Also, i wasn't able to find other libraries that provide Adaboost implementations (in python) for classification.

If customizing sklearn is the good way, i'd appreciate some guidance about where to start from (API documentation is not that clear).

ps: i am relatively new to this area, but unfortunately this is a task that was given to me and it's really frustrating that i can't find a way to accomplish it.


texts:
I asked the same question at Cross Validated, here
I implemented a K-Medoid clustering algorithm recently; I have a number of points $x_1, ..., x_n$ which have various properties and a distance function $d$ that maps two points to some nonnegative number. The clustering works fine (with some number $k$ of cluster medoids where $k$ is a lot smaller than $n$), but I wanted to be able to change the clustering according to a property of my points, more specifically I'd like more clusters to appear where my points have a high value in some property. I tried scaling the distance matrix like this:

$newd (x_i, x_j) = p(x_i) * d(x_i, x_j)$
where $p(x)$ is the value of the property of $x$.

My expected result was to see more clusters around points with a high value of $p$, but instead, I get the exact same clusters. I'm probably missing something very basic here.
So, to sum up, my question: I'd like to see more clusters around points with a certain property. I assume I can achieve this by changing my distance function, but I don't quite understand how to do it?


texts: Efficient clustering of sparse binary vectors

texts:
I'm trying to perform clustering of the data to improve the efficiency of brute-force kNN . The dataset consists of objects described by a large set of binary features, each identified by a 32-bit hash code. Data point can be understood as a 2^32 elements long very sparse binary vector, with bits set to 1 on the positions denoted by hash code of feature. To simplify, each of the data points is being represented as an array of hashes - if we know which of the bits are set to 1, we know which of them are set to 0.

I have a working distance function (mentioned in here) but have difficulties to cluster the dataset in reasonable time.  Because of the binary nature of the data it is impossible to create any kind of mean value based on a collection of datapoints, so k-Centroids is not an option. I tried k-Clustroids but it doesn't converge, hierarchical approaches are too inefficient. Do you happen to know any clustering algorithm that would efficiently handle fixed size dataset, with custom metric calculation method without the need to create any temporary, centroid datapoints?

Many thanks.


texts: Logistic regression with high cardinality categorical variable

texts:
I have a logistic regression model where I care about predictive power solely over comprehensibility. I'm interested in predicting win rates in a video game.

There are 133 characters. Each team picks 5 of them (no repeats). Each of these characters is assigned to one of five positions (again no repeats).

Currently I have each of these characters as a dummy variable. In addition I have an interaction variable between each of these variables. The position of a character is not included in the model at present.

I know I can trim down the size of the model by excluding low-playrate characters, however my concern is that the required sample size is still far too small for the complexity of the model. Any advice would be appreciated.

Sample Size: Aprox. Two million


texts: export R neuralnet package model to PMML

texts:
when I look into the R package pmml, I found that, it is possible to directly export nnet model into PMML format.

But actually I am using neuralnet package since it has more features and flexibility.

The problem is:

How can I export neuralnet model to PMML format, I checked pmml package, it doesn't work.

Any other options like transform neuralnet model to nnet model, or some other PMML packages to use? thanks


texts: Very low accuracy of new data compared to validation data

texts:
I'm trying to train the neural network to predict the movement of a particular security on the market.

I teach on historical data collected for the year. At the entrance of the neural network candlesticks are served: close price and value

Before submitting, these data are normalized separately for each dataset. This happens with the z Score algorithm. Then the question immediately arises... output can not be obtained in the limit [0;1] or [-1;1] and can reach up to 10 or more (in both directions). Is that okay?

All sets are first shuffled, and then divided into two parts:: train and test (80:20)
The output is a class: the price will go up or down.

For train, the error is reduced to zero and the accuracy to 100%. For test, the error reaches a minimum of 0.28 and the accuracy reaches 90%

I tried to test this neural network to predict the next two months, which were not used in this neural network. They went through exactly the same normalization. However, the accuracy of this forecast is 0.5289256204750912. And the error - 1.8223290671986982

I know I'm doing something wrong, but I don't know what... I hope someone can help me figure this out.
PS: ZScore was used To the usual min, max. However, this particular success was not observed.


texts: Simple Linear Regression-----How to make my model more efficient??

texts:
I am working on a simple linear regression model,

  This is my Python code :

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

dataset=pd.read_csv('sample.csv')
X=dataset.iloc[:,:-1].values
Y=dataset.iloc[:,1].values

from sklearn.model_selection import train_test_split
X_train,X_test,Y_train,Y_test=train_test_split(X,Y,test_size=1/3)

from sklearn.linear_model import LinearRegression
regressor=LinearRegression()
regressor.fit(X_train,Y_train)

plt.scatter(X_train,Y_train,color='red')
plt.plot(X_train,regressor.predict(X_train),color='blue')
plt.title('X vs Y(Training Set)')
plt.xlabel('X')
plt.ylabel('Y')
plt.show()

plt.scatter(X_test,Y_test,color='red')
plt.plot(X_train,regressor.predict(X_train),color='blue')
plt.title('X vs Y(Test Set)')
plt.xlabel('X')[enter image description here][1]
plt.ylabel('Y')
plt.show()`

  This is my plot of Training Set Training Set
  This is my plot of Test Set Test Set

How can I increase the efficiency of my ML model???
This is my first ML model, so all suggestions are welcome.
Thanks in Advance


texts: model to predict annual outcome based on previous years data

texts:
I have below datasets for two years each holding about 10.000 records. Every week a new report is generated that shows the performance for the current or any previous month. Therefore a more recent weekly report would overwrite the monthly value of a previous report (for that specific month). I need to predict this year's performance based on the weekly data coming in.

I have two questions:

How would I need to structure my dataset for machine learning? If I
keep the above I would get about 150 features (3 per week) and I do not
know whether there is even a ML algorithm that can handle the many n/a (all
forthcoming weeks) for the year for which I need to run the
prediction. I can also transform this into a new dataset by combining
the three columns by week into three features (would result in long
narrow table). But that would replicate each entity and output value about 50 times (once per
week).
Based on the answer to 1. what would be suitable training
algorithms?


texts: Real distance between bounding box centers

texts:
Assume that I have a camera pointing in a specific direction. I know the Euclidean distance (Real world distance) of the camera to a fixed point, X (mm). Using YOLOv3 I have detected all the occurences of an object and have the bounding boxes and their centers. I also am able to determine the Euclidean distance of the camera to all bounding box centers. I have the availability to the following measures:
focal length,
image height (px), ## Photo
image width (px),  ## Photo
Real object height (mm),
Real object width (mm),
sensor height (mm),
sensor width (mm),
pixel size ($mu$m),
Field of view (vertical),
Field of view (horizontal)

and the following calculated measures:
0. The pixel-coordinates of every single bounding box center.
1. Euclidean distance from camera to all instances of a specific and detected object
2. object height and width (in pixels) # used to calculate 1.
3. Angle made by between every pair of bounding box centers relative a fixed point on the image (actually, an epipole determined by the use of a second image)

Now, what I would like to do is to calculate the Euclidean distance (Real world distance) between the objects in the photo. I naively thought that knowing the pixel distance and the pixel size woul be enough but apparently I was mistaken.
Any insight would be greatly appreciated.
Thanks!


texts: How can I use variable length inputs to train a regression model?

texts:
I'm working predicting a value $y \in \mathbb{R}$ from the value of $x_{n+1}$, where $n$ is the number of samples ($x_{i \in [1,n]}$) used for training.

Each training sample $x_{i}$ is a time series of variable lengths. How can I engineer features to predict $y$ while making the most of the samples available?

Nota bene:

I'm working with Python & Scikit-learn
the length of the time series may be considered as an explanatory variable

Thanks for your help :)


texts:
I'm thinking about a problem where I have data on different A/B tests and which variant won.
The features would be a bunch of one-hot encoded functionalities of each page (i.e., blue button, square button, video on the right side, etc).
I want to see which features are most important to predicting whether a variant has won.
Some tests can have multiple features (multivariate testing).
Does anyone have any ideas, giving this framework, to find out which features or combination of features are most important to figuring out what functionalities on a page are most important to predicting a winning variant?
In a test where you are testing multiple things (such as button size, button color, and text). How do testers usually tell which of the changes are most important?


texts: Pairwise 3D object correlation between 2 objects

texts:
I have a dataset which contains 3D CT scans from different patients along with the segmenation masks of a certain organ. The 3D scans have been drawn each day for a period of 30 days for each patient. I want to conduct a series of experiments and measure the similarity of the organ for each patient over this period. There is no need to tackle this dataset as a whole, all I need is to find the pairwise correlation of the organ from the first day and on.
Is there any technique to find the correlation between two 3D objects?


texts: What is External representation of time in Sequential learning?

texts:
I am reading the literature on sequential learning and it is often mention that in order to handle sequential/temporal data, there two categories of work in sequential learning

External Representation of Time
Internal Representation of Time

I am not able to understand the difference between two, for example in the paragraphs below, it is hinted that external representation can be inserted directly by preprocessing time through a time to space transformation. Following example is given to elaborate this.

For instance, one effective way of constructing a spatial representation of temporally-
occurring information (which is not only limited to neural computing) is to create
the power spectrum of the incoming information and use it as a static input image.

Is there any simple and intuitive example to understand this? I cant get time to space transformation or power spectrum thing.


texts: Algorithms for Anomaly Detection of Event Sequence Data [Python/R]

texts:
I am building an anomaly detection system of event sequence data (transactions). For each timestep, a transaction can be in any of 76 different stages.
My dataset is therefore a 3D array of size(m,t,N).
m is number of samples (~40 000), t = the length of the longest transaction (I am zero padding all other sequences of shorter length), N is the number of possible events at each timestep. $x_{m,t,N} = \{0,1\}$ where $x_{v,u,w} = 1$ if transaction v is in event w at timestep u otherwise it's 0.
So far I have created an LSTM Variational Auto Encoder in Python with Keras and doing outlier detection by applying Local Outlifer Factor on the latent representations of the transactions. It's based on this paper https://arxiv.org/abs/1906.10896 and following various blogs such as https://blog.keras.io/building-autoencoders-in-keras.html.
I would like to compare the results with other techniques. I was wondering if anyone has any other suggestions on algorithms and techniques for anomaly detection in unlabeled event sequence data?


texts:
The paper explaining 'Advantage Updating' as a method to improve Q-learning uses the following as its motivation.

  Q-learning requires relatively little computation per update, but it is useful to consider how the number of updates required scales with noise or with the duration of a time step, At. An important consideration is the
  relationship between Q values for the same state, and between Q values for the same action. The Q values Q(x,u1) and Q(x,u2) represent the long-term
  reinforcement received when starting in state x and performing action u1 or u2 respectively, followed by optimal actions thereafter. In a typical reinforcement learning problem with continuous states and actions, it
  is frequently the case that performing one wrong action in a long sequence of optimal actions will have little effect on the total reinforcement. In such a case, Q(x,u1) and Q(x,u2) will have relatively close values. On the other hand, the values of widely separated states will typically not be close to each other. Therefore Q(x1,u) and Q(x2.u) may differ greatly for some choices of x1 and x2. Therefore, if the network representing the Q function makes even small errors, the policy derived from it will have large errors. As the time step duration dt approaches zero, the penalty for one wrong action in a sequence decreases, the Q values for different actions in a given state become closer, and the implied policy becomes even more sensitive to noise or function approximation error. In the limit, for continuous time, the Q function contains no information about the policy.
  Therefore, Q-learning would be expected to learn slowly when the time steps are of short duration, due to the sensitivity to errors, and it is incapable of learning in continuous time. This problem is not a property of
  any particular function approximation system; rather, it is inherent in the definition of Q values.

How do I mathematically prove that this effect occurs?


texts: Latent space for cross domain numerical features

texts:
I would like to find the shared latent space between two set of features. I have source and target domain features already extracted from images. I have 4 set of feature vectors for normal and abnormal source and target domains. I would like to train on normal source and target features and predict on abnormal sets. How do I that?
I have this idea, that if I create a shared space between two domains and give it to a classifier, it will predict if shared space for testing phase is different, then it is abnormal. I have read some literature for images, but I am no sure how to do it for numerical features?


texts: Which definition of Likelihood function is correct?

texts:
In the online version of the Deep Learning book on chapter 5 the estimator for likelihood function is defined as:

That is the product of individual probabilities. After taking the log it arrives at the log-likelihood funciton (Eq.1):

It then rescales the above by dividing it by m to obtain a version expressed as the expected value (Eq.2):

OK. Here is my first question:
The expected value is defined as

And so I think the expected value of log will be the below expression which is not the same as Eq.1 in the book.

The only way for this to be true(that is to divide the Eq.1 by m and arrive at Eq.2 as claimed by the book author) is for the probabilities of p(data) to be uniform. But this is also not a valid assumption.
So I don't see what I am missing here?

Next, the book argues that maximizing the above log-likelihood function (Eq.2) is same as minimizing the KL divergence:

Or more simply just minimizing the second term. And so the author says that either way we arrive at the same function as Eq.2.

On the other hand, from the Wikipedia page the cross entropy of two probability is defined as :

I can understand this definition as the expected value of log(q) which is same as the expression in the Eq.2. but not Eq.1
From the same Wiki page the likelihood definition is given as below which is different than the likelihood function definition from the book(above). Here the probability of q (model) has been raised to the number of occurrences; which then on taking log it is understandable to see it as the expected value.

So I am confused that first of all which definition of likelihood function is the correct one?
Given the definition from the Wikipedia I can understand that maximizing the log-likelihood function is same as minimizing the cross-entropy function. However I cannot arrive at the same conclusion from the definition of the likelihood and the log-likelihood function given in the book, for the reasons I explained above.


texts: Is a data set considered to be linearly separable if it can only be separated by multiple hyperplanes?

texts:
For example, on the linear separability Wikipedia article, the following example is given:

They say "The following example would need two straight lines and thus is not linearly separable".

On the other hand, in Bishop's 'Pattern Recognition and Machine Learning' book, he says "Data sets whose classes can be separated exactly by linear decision surfaces are said to be linearly separable".

Under Bishop's definition of linear separability, I think the Wikipedia example would be linearly separable, even though the author of this Wikipedia article says otherwise. This is because Bishop says that we can use multiple linear decision surfaces (hyperplanes) to separate the classes, and it's still considered to be linearly separable data. Bishop implies this by referring to linear decision surfaces, plural, not singular.

Logically, I agree with Bishop. After all, the classes in the Wikipedia example are being separated by linear decision surfaces. So how can one then turn around and say the data set isn't linearly separable? Well, perhaps you could enforce the rule that a data set is only linearly separable if we can separate $N$ classes with $N-1$ decision surfaces. But why would you define linear separability in this way?

So, is the Wikipedia example linearly separable or not?


texts: Suitable CPU with RTX3070 GPU for deep learning

texts:
I am trying to get a GPU for my deep learning research. Due to stock limitation, I can only get a RTX3070. In my lab, there's some PC/workstation whereby I can install the RTX3070. However, I wonder if the CPU is suitable. Will it cause bottleneck with the GPU. I used the bottleneck calculator to check.
but it's more suitable for games. They mention that 1 and 3 below are not suitable but I'm not sure how accurate it is.
Does anyone have experience how to check?
The CPUs I have are:

i7-4770@3.4ghz PC 32gb ram
xeon 4216@2.1ghz 192 ram
xeon e5520@2.27ghz 64gb ram

RAM, PSU and SSD can be upgraded so it's not a big problem. The main task is deep learning.


texts:
In my use-case of multi-class classification, my data distribution is like below:

It might be too silly to ask this (and possibly could be gravely wrong), but is there a provision to default an o/p class to a value which is safe to be defaulted than to be predicted a completely wrong outcome.

Ex. Suppose the case, where an incoming email meant to be for "hardware" department but is predicted as for "Company Leadership" department and hence routed to all senior members of the company. In such case, since the prediction accuracy of the entire output class is below say 20% accuracy, I would like to default that entire class to "service desk" group and let them manually sort it.

Hope I made my question clear (Might be confusing as well!). Please let me know if any clarifications required. I would be happy to amend the wordings.

Thanks. :)


texts: Named Entity Recognition using context of the sentence

texts:
I have a problem in which I want to know how can we extract or name the entity based on the context in which it is getting used in a sentence.

For example:
If we have to extract date field which is used in the context of the date of birth only then how can we do that.

I know that we can use regular expression, spacy, NLTK to extract date field from a document. But I am unable to determine the approach to extract date based on the context in which it is getting used.


texts: Why are teacher forcing and non teacher forcing test-losses are negatively correlated?

texts:
I'm trying to train a recurrent neural network (more specifically an LSTM) to learn to predict a certain time series. The time series has 10 variables with datapoints every 5 minutes and I have data over a period of 50 years.
I initially trained the network with teacher forcing using the following step loss function (using pytorch):
def train_step_tf(input_seq, target_seq, model):
    output, hidden = model(input_seq)
    loss = mse(output, target_seq)

    return loss

For all time steps $i \in \{1, \ldots, \text{len(input_seq)}\}$, this feeds data $x_i$ into the model and computes the mse los between $\hat x_{i+1}$ and $x_{i+1}$.
Unfortunately the prediction performance (feeding in a 100 step primer and then predicting the next 100 steps) was very low. I assumed this was because when training with teacher forcing, the network only every sees true timeseries data as input and gets confused when it sees it's own slightly atypical generated data.
I then trained it without teacher forcing using the following step loss:
def train_step_ntf(input_seq, target_seq, model, primer_length):
    primer_inp = input_seq[:, :primer_length, :]

    _, hidden = model(primer_inp)
    hidden = hidden.detach()

    train_inp = input_seq[:, primer_length:, :]
    train_targ = target_seq[:, primer_length:, :]

    input = train_inp[:, 0, :].unsqueeze(1)
    loss = 0

    for c in range(train_inp.shape[1]):
        output, hidden = model(input, hidden)
        loss += mse(output, train_targ[:, c, :].unsqueeze(1))
        input = output.detach()

    return loss/train_inp.shape[1]

Unfortunately the predictive performance is still very low.
I then trained various network configurations with and without teacher forcing (at train time) and regularly logging both the tf and non-tf test-set loss.
As I understand it, training with teacher forcing should be faster, but not using teacher forcing could allow for better predictive performance. Overall both losses should be equivalent however, in the sense that when the teacher forcing loss goes to zero (meaning the network can predict the next step 100% accurately) the non-tf loss should go to zero as well (predicting the next 100 steps all at once), and vice versa.
Unfortunately when plotting the final tf test loss and non-tf test loss for each run, it seems like they are negatively correlated:

Color indicates whether I used teacher forcing at train time. Of course when training with teacher forcing the teacher forcing test loss is generally better. But weirdly the better a runs teacher forcing test loss was, the worse was the non-tf test loss. Can anyone explain why that is? Thanks


texts: What interactive graph package do you use for timeseries data?

texts:
I have millions of sensor readings taken over time.  I'd like to create an interactive plot in Google colab where I can easily zoom in on sections.  I've done something similar with the Plotly package.  However, it fails when the amount of data gets over ~ 700,000.

Are there interactive plotting packages that work with Google colab and handle millions of data points?

Thank you.


texts: Images Dataset Augmentation with fixed parameters of Crop & Rotation Angle

texts:
I have a Data Set of 170 images in total. of each image i want to create 36 images like this.

An image is cropped from its top-left corner by cropping 5 percent from the top and from the left side of the image. Then, the new image is scaled back to the original size of 188x188. This cropping method could be also done from three other corners. The same thing could be done with 10 percent cropping. This results in 8 images besides the original image. Afterward, these 9 images are rotated by 0, 90, 180 and 270 degrees. Therefore, 36 synthesized images are derived from each input image.

I need python code for this dataset augmentation.


texts: Is it logical to calculate zscore for list A using the standard deviation and mean from list B?

texts:
I am working on an application where we collect the total income of a household. We have a $3^{rd}$ party integration which provides us the household income estimates. I have calculated standard deviation and mean for both these datasets. To compare how many standard deviations away the $3^{rd}$ party integration data is from my collected income (or vice-versa), I am planning on calculating the z-score as follows:
z-score = (list_income_collected - mean_of_3rd_party_income_estimates)/
           standard_deviation_of_3rd_party_income_estimates

Is it logically sound?


texts: Why do RNNs usually have fewer hidden layers than CNNs?

texts:
CNNs can have hundreds of hidden layers and since they are often used with image data, having many layers captures more complexity.

However, as far as I have seen, RNNs usually have few layers e.g. 2-4. For example, for electrocardiogram (ECG) classification, I've seen papers use LSTMs with 4 layers and CNNs with 10-15 layers with similar results.

Is this because RNNs/LSTMs are harder to train if they are deeper (due to gradient vanishing problems) or because RNNs/LSTMs tend to overfit sequential data fast?


texts: Correlating activity between entities using Python

texts:
I have some data that contains IDs along with an associated column containing times. What I want to do is to be able to determine which IDs are most similar, based upon the times e.g. Which IDs have correlated times for performing an action

The data looks something like this;

ID      Time(secs)
AAAA    1
AAAA    6
AAAA    5
AAAA    2
AAAA    4
BBBB    2
BBBB    4
BBBB    6
BBBB    3
CCCC    3
CCCC    4
CCCC    1
CCCC    6
DDDD    7
DDDD    4
DDDD    5
DDDD    3

Naively I initially thought this would be a simple case of plotting the values and then calculating the Correlation Coefficient but I soon realised this isn't possible;

ID             CorrCoef
AAAA>BBBB      ????
AAAA>CCCC
AAAA>DDDD
BBBB>CCCC
BBBB>DDDD
CCCC>DDDD

I am now thinking I need to be able to compare 2 populations using something like a t-statistic or perhaps by using something like an AutoRegression. It is fair to say I am now feeling a little bit out of my depth with something that seemed quite basic initially.

Does anyone have any pointers as to the best way of doing this?

Thanks in advance!

Edit

Based upon a suggestion below, it seems a KS test could be useful here. What I need to do therefore is subset the data so that it can be fed into the below line of code

scipy.stats.ks_2samp(data1, data2)


texts: Is there any measure to find how much classifiers are confident?

texts: Assume that we have two classifiers (C1, C2) and two classes (A, B). These classifiers give us the belonging probability for each class for each instance. Suppose that we have an instance X which is actually should be classified as A. C1 classification result is (1, 0) and C2 classification result is (0.9, 0.1) which means they both classified X correctly as A. Obviously C1 is more confident. Is there any measure that I can use to compare my classifiers based on that?


texts: word2vec - log in the objective softmax function

texts:
I'm reading a TensorFlow tutorial on Word2Vec models and got confused with the objective function. The base softmax function is the following:

$P(w_t|h) = softmax(score(w_t, h) = \frac{exp[score(w_t, h)]}{\Sigma_v exp[score(w',h)]}$, where $score$ computes the compatibility of word $w_t$ with the context $h$ (a dot product is commonly used). We train this model by maximizing its log-likelihood on the training set, i.e. by maximizing $ J_{ML} = \log P(w_t|h) = score(w_t,h) - \log \bigl(\Sigma_v exp[score(w',h)\bigr)$

But why $\log$ disappeared from the $score(w_t,h)$ term?


texts: Can a neural network recognize a letter B as an A if your trained it so?

texts:
You have a neural network. And you have, say, pictures of $100,000$ hand-written letters (A-Z). Now you make a typical Training and the neural network will recognize an A as an A, a B as a B, ...

Now you repeat Training but with a change: you tell the neural network for all pictures with the letter B that its an A too.

Now the questions after training:

Does the network recognize A as an A as good as before?
Does the network recognize B as an A too?